Defining Artifact Identity and Safe Cache Reuse in Android CI

Morning, everyone. Leo and Amira have joined from Build Infrastructure, Daeho is representing System Quality, and Iris is here with me from Platform Integration. We need to discuss a contradiction in yesterday's Android integration run. CI marked candidate C61 as qualified, but a manual check found that its vendor service manifest did not match the tested image. Nobody has reported a device failure from this candidate, and it has not left the lab. Nevertheless, we cannot explain why the dashboard says the exact candidate passed. Today we must establish what was actually built and tested before we design a faster pipeline.

I found the mismatch while investigating a missing service declaration, not while chasing a failed test. The report attached to C61 says all forty-eight interface checks passed. Its attachment contains the service list from C58, where that declaration still existed. The timestamps are plausible, and the report names the current branch, so a routine review would probably accept it. Please treat this as an evidence-integrity problem. A passing result belongs to an input set, not to a branch name. Until we identify that input set, I cannot use the forty-eight checks to qualify C61.

The shared cache is the obvious suspect, but there are two separate caches in this pipeline. One stores compiled objects; the other stores completed test reports. Object reuse can be correct even when a report must be regenerated. Our recent optimization shortened the report key to the framework revision and test suite version. It omitted the vendor image digest because the interface team described those tests as framework-only. That description was incomplete: the suite reads the vendor manifest at runtime. I need to check whether the wrong report came from this key or from a later attachment step.

I pulled the cache access records before joining. C61 requested a report key that already existed, and the cache returned the C58 entry. The job wrapper then rewrote the display label to C61 without changing the embedded manifest. That explains the misleading branch and timestamp, but it does not establish that every cached report is wrong. We have eleven candidates since this optimization went live. I can reconstruct which entries they consumed from the immutable access log. I would like to preserve those entries before anyone flushes the cache, because deletion would remove evidence we still need.

Platform Integration has a practical concern. If we disable every cache immediately, the candidate queue will grow from roughly three hours to seven, and four teams will lose their daily integration window. I agree the report cannot qualify C61, but I do not support treating all compilation reuse as unsafe. Can we bypass report reuse while keeping object caching, then regenerate qualification for the affected candidates? That would contain the evidence problem without starving developers of builds. I also want a visible label for an unqualified image, so engineers do not mistake availability for permission to promote it.

That distinction is useful. Let us follow one candidate from source checkout through image assembly, test execution, and report publication. At each boundary, state which identifier is authoritative and whether it can be rewritten. A source commit alone does not identify a device image if vendor blobs, configuration, or the toolchain change independently. We need an input manifest that survives the entire chain. Leo, please show the current key. Daeho, then list what the interface suite actually reads. I want us to discover dependencies from behavior rather than rely on the name of the test.

The report key currently concatenates the framework commit, suite revision, and target product string. It does not include the vendor digest, product configuration hash, boot image digest, or execution mode. The target product string is stable across two hardware revisions with different service manifests. That was acceptable for selecting build rules, but it was a poor identity for a test input. I propose separating the human-readable product label from an immutable candidate identifier. The latter would be derived from a canonical input manifest and printed in the image, job metadata, and result envelope.

The suite reads more than the vendor manifest. It queries service availability from the running device, compares interface hashes, inspects declared transport, and checks a policy file from the product partition. The execution mode also matters because recovery mode exposes a smaller service set. A digest of only the vendor partition would still miss those inputs. I suggest classifying reports by declared dependencies, with a default of no reuse when a suite has not published that declaration. We should not ask infrastructure to guess the dependencies of tests owned by another team.

There is another boundary we should inspect: the device provisioning worker. It selects images by a mutable alias called latest-qualified. Yesterday that alias moved while a test job was waiting for a device. The report cache caused the C61 mismatch we saw, but an uncached job could still flash a different candidate if it resolves the alias at execution time. I can change the handoff to carry exact image digests instead. Before a test starts, the worker should read back the flashed identifiers and compare them with the request. A mismatch should prevent execution, not merely add a warning to the report.

I agree with exact digests, provided developers can still choose a friendly alias when requesting a build. The system can resolve it once at submission and show both the label and frozen identifier. Otherwise people will copy long hashes incorrectly or invent their own shortcuts. We also need to handle a device whose bootloader cannot expose every partition digest. Could the provisioning worker verify the downloaded package and a signed build manifest reported by the device? I do not want a check that is impossible on half our lab fleet, because it would quickly become a routinely ignored warning.

Yes, but we must distinguish package provenance from device state. A verified package proves what we intended to flash; a device-reported manifest proves what the running image claims to contain. Neither alone catches a provisioning script that skips one partition. For capable devices we can compare partition digests directly. For the older fleet, we should require a clean provisioning path plus package verification and manifest agreement, and record the weaker assurance level. I would keep those results useful for development but prevent them from satisfying a release gate that requires full image identity.

My concern is that dependency declarations will become another stale file. A test owner may add a product-policy read and forget to update the list. We need a way to challenge the declaration automatically. For each reusable suite, change one input that it says is irrelevant and check whether observed output changes. That is not a complete proof, but it can expose obvious omissions. More importantly, qualification suites that observe running services should execute against every candidate. The savings from report reuse are small compared with the cost of a false qualification, so I would exclude this interface suite entirely.

I support that for release qualification, but I would preserve a different development signal. Engineers often need to know whether a framework-only edit breaks a host-side parser before waiting for a device. A cached host test with a complete source and toolchain key can still answer that question. The dashboard should show host checks and device qualification in separate columns, not collapse them into one green badge. If device tests are pending, the candidate can remain available for debugging while promotion is disabled. That gives people useful feedback without claiming more evidence than we have.

Let me put numbers around the trade-off. Report reuse saves twenty-two minutes per candidate. Object caching saves two hours and forty minutes. Turning off report reuse for the qualification suite therefore has a manageable cost, while disabling object reuse would disrupt the queue substantially. We can run one uncached build each night as a comparison against cached compilation. If image digests differ, we compare the manifests and investigate nondeterministic output before blaming the cache. The comparison should use identical inputs and toolchains; otherwise two different images tell us nothing about cache correctness.

We have a workable boundary now. Keep object caching, allow explicitly keyed host-result reuse, and prohibit report reuse for device qualification. Before accepting that decision, I want to address what happens when a cache server is unavailable. The build should fall back to compilation rather than fail simply because an optimization is missing. However, an unavailable provenance record is different: without the input identity, promotion must remain blocked. We should write those two behaviors separately in the pipeline contract. A slow build is inconvenient; an unexplained pass undermines every downstream decision.

Here are the actions for today, October fourteenth. Leo will disable qualification-report reuse and remove label rewriting by noon Seoul time. Amira will preserve the eleven candidates' cache records and publish an input-to-report map by two this afternoon. Daeho will rerun the interface suite on C61 using digest-pinned provisioning and report by five. Iris will update the dashboard so available, host-checked, and device-qualified are separate states by tomorrow noon. Until that work is verified, none of the eleven candidates can be promoted using the cached interface result. Other valid evidence remains attached with its original identity.

I can meet noon for the immediate containment. For the permanent key design, I need until Friday, October sixteenth, at three in the afternoon. The manifest will include source revisions, vendor and boot image digests, product configuration, toolchain identity, suite revision, and execution mode. Host suites can declare a narrower subset, but the declaration will be versioned with the suite and reviewed by its owner. Report publication will preserve the original candidate identifier and state whether a result was executed or reused. A copied report will never receive a new execution timestamp.

I will preserve both successful and failed cache lookups, including the requested key, returned entry, publishing job, and attachment digest. The map will identify candidates with definitely mismatched reports, candidates with matching inputs, and candidates whose records are incomplete. I will not mark an incomplete candidate as safe simply because no mismatch is visible. For digest-pinned provisioning, I can have a prototype ready tomorrow at four. It will resolve aliases at submission, verify package bytes before flashing, and record device assurance level after boot. Daeho can review the result envelope before we enable it broadly.

For C61, I will use a device that supports partition-digest verification and preserve its provisioning log. The rerun will produce a fresh report even if every test passes exactly as before. We must also retain the original cached report, clearly marked as invalid for qualification, so reviewers can follow the correction. By tomorrow morning I will provide a prioritized retest list for the remaining ten candidates based on the input map. Matching inputs may justify retaining some host evidence, but device qualification will still require an executed result. I will make that distinction explicit in the review notes.

I will change the candidate page and the promotion service together. A visual badge alone is insufficient if an API can still promote an unqualified image. The service will require an executed device report whose candidate identifier matches the selected image and whose assurance level meets the release policy. Development downloads remain available with a clear status. I will test the policy using a deliberately mismatched report, a reused host report, and a correctly executed device report. We should see the first two denied for promotion and the third accepted, with understandable reasons in the audit log.

Our follow-up is Friday, October sixteenth, at four thirty in the afternoon Seoul time. We will inspect the input map, C61's fresh qualification, provisioning checks, and the promotion policy demonstration. Success means a candidate cannot inherit another image's qualification even when the framework revision is unchanged. We will also confirm that cache failure only slows compilation and that missing provenance blocks promotion. This is a pipeline contract review, not a demand that every cache entry be deleted. If the contract is incomplete, we keep the affected candidates restricted and assign the missing work with a new deadline.

For the demonstration I will submit two builds with the same framework commit but different vendor manifests. They must receive different candidate identifiers and independent device reports. I will then repeat a host parser test with identical declared inputs to show that valid reuse still works. Finally, I will disconnect the object-cache endpoint and demonstrate compilation fallback. Those cases exercise the boundary we agreed rather than simply showing a happy path. I will include one malformed dependency declaration to confirm the pipeline rejects it with an actionable message instead of silently broadening the key.

I will prepare a small reviewer checklist focused on identity and evidence. It will ask which image ran, which suite ran, whether the result was executed or reused, and what device-state verification supports it. Reviewers should not need to search three systems to answer those questions. I will measure the time needed to find each answer on the new page, using two engineers who did not attend this meeting. If they cannot resolve the candidate identity within two minutes, we should improve the presentation before calling the process complete. Clear evidence should reduce review effort, not merely add more fields.

There is one operational detail: preserved cache entries consume space, and our retention job normally removes them after seven days. I will place the eleven candidates' relevant entries under a temporary investigation hold until October thirtieth. After the review confirms that their records are archived in the issue, we can release the hold through the normal retention process. The hold will be limited to the affected keys, so it will not fill the whole cache. I will also record who can release it; otherwise a cleanup script could erase the evidence while another team is still reviewing the correction.

That covers my scheduling concern. Developers will still receive cached compilation and useful host feedback, while promotion waits for evidence tied to the actual image. I will notify the integration owners today that C61 is downloadable for investigation but not qualified for promotion. The notice will link the input map and fresh rerun when they are available, rather than blame a team or call every past result unreliable. For Friday's review I will bring the API denial cases and a screenshot of each candidate state. With those checks, we can make the fast path understandable and the qualification path defensible.