A cloud retirement notice arrived with a simple instruction: move the application from Node 22 to Node 24 before support ended. Seven days later, the platform had a new authoritative API service, a measured production capacity envelope, three exact-SHA release tracks, a stopped-but-preserved legacy resource, and a codebase that no longer taught future engineers—or future AI agents—the wrong architecture.
This is the end-to-end account of that migration. It is also a case study in a newer operating model: a human technology owner sets direction, constraints and irreversible decisions; an AI coding agent executes the bulk of a real enterprise workstream autonomously; an independent model challenges the evidence; and production—not persuasive prose—decides whether the work is complete.
The result: Codex took a Node runtime retirement problem and carried it through architecture, repository refactoring, infrastructure provisioning, CI/CD, deployed data-plane testing, traffic cutover, live business ETL soak, low-cost telemetry, legacy-host deactivation, source-code de-Functionization, documentation and outage recovery. I remained accountable for strategy, risk appetite, cost and production authority.
Names, URLs, subscription identifiers, credentials, business-unit identifiers and commercially sensitive data are omitted or replaced with generic labels such as [WEB APP], [PLATFORM API], [AI SERVICE] and [RETAINED FUNCTION]. The architecture, engineering failures, decisions and measurements are real.
The email was the trigger, not the problem
The notice said that Node 22 LTS support on the cloud application platform would end in April 2027. Two production applications were called out: the web application and a separate AI service. Those were straightforward runtime upgrades and were handled first.
The awkward component was the business API. It still lived on a serverless Functions runtime. Upgrading it was not merely changing a version string. The provider roadmap was pushing serverless Linux workloads toward a different hosting model, while the existing API already carried the complexity of function registration, packaging, network allow-listing, shared code mirrors, deployment-specific dependencies and a local development path that no longer resembled the rest of the platform.
At the same time, the platform was growing. A governed product-pricing capability was being built for future consumption by another internal application. That made the API an architectural product, not an incidental set of Functions.
I made the strategic call: create a dedicated backend API service on the same small, fixed-cost Linux tier already proven by the AI service. Keep the AI runtime separate. Move the business APIs away from serverless. Make Node 24 a common platform rule. Preserve the old cloud resource until the new path had earned trust.
The target architecture
The final topology is intentionally boring. That is a compliment. Users enter through one authenticated web front door. The web service proxies business API traffic over a shared-trust boundary to a dedicated Node 24 API. The AI service remains independently deployed. Document storage, object storage and embedded ETL stay behind purpose-specific credentials. The former Function resource exists, but it is stopped and outside every supported request, deployment and rollback path.
| Component | Responsibility after migration | Deliberate boundary |
|---|---|---|
| Web App | Authenticated user experience, web delivery, embedded ETL orchestration and trusted API proxy. | The browser does not receive internal service credentials. |
| Platform API | Authoritative Node 24 business API, route registry, authorization, health, telemetry and data access. | Public ingress fails closed; the web service is the trusted caller. |
| AI Service | Longer-running AI orchestration with its own dependency and scaling profile. | AI is not folded back into the general API merely for symmetry. |
| Data plane | Application documents, ETL files, operational logs and published artifacts. | Credentials are purpose-specific; control-plane ownership is not mistaken for data-plane access. |
| Legacy Function resource | Preserved cloud infrastructure that may be reconsidered for an unrelated future feature. | It is not authoritative, monitored, deployed, documented or supported as a compatibility API. |
The key sentence is the last one. Keeping a resource is not the same as keeping an architecture. The cloud object survived; the obsolete business-API contract did not.
The operating model: autonomy inside explicit authority
I did not give Codex an open-ended instruction to “migrate the backend.” I established a production maintenance workstream with a durable plan, required the repository’s incident-derived contracts to be read first, and defined what the agent could do autonomously versus what required an owner decision.
| Codex executed autonomously | I retained as owner | Independent review challenged |
|---|---|---|
| Repository and live read-only inventory | Dedicated service versus continued serverless investment | Whether evidence proved the deployed context or only a local approximation |
| Architecture and phased implementation plan | Cost ceiling and initial service tier | Package-boundary, authentication and cleanup assumptions |
| Adapters, server, route registry, tests and packaging | When to move production traffic | Same-SHA races, diagnostic quality and drift protection |
| Cloud provisioning, deployment workflows and verification scripts | How long to soak and which live business workflows mattered | Whether warning-only dependencies were incorrectly blocking release |
| Browser testing, telemetry, failure investigation and daily monitoring | Stop versus delete the legacy resource | Whether credentials were being re-derived instead of threaded through |
| Source cleanup, documentation sweep and exact-SHA outage recovery | Approval to merge, push and mutate production | Whether future agents could accidentally resurrect the retired pattern |
The agent kept the implementation plan as an engineering ledger. It recorded completed actions, failed proofs, rejected designs, observed production state and remaining gates. That was more valuable than a static project plan because it made memory durable across a week-long task and gave the reviewer a common evidence base.
My role did not shrink; it moved up the stack. I spent less time transporting commands and more time deciding architecture, acceptable blast radius, operational evidence, cost posture and when the system had earned the right to advance.
Phase 0: discover the system that actually exists
The first autonomous task was not coding. Codex read the application contract, runtime rules, architecture and deployment documentation, then traced the implementation and live cloud configuration.
That inventory found an API surface of 96 HTTP routes at the time, zero registered timers, shared libraries consumed by the web and AI deployment artifacts, a Function-specific adapter and registration facade, purpose-specific and fallback storage credentials, a web-only ingress requirement, and a local development workflow still shaped around the old host.
The agent generated a route manifest rather than trusting a number typed into a document. Name, method, path and ordering became a machine-checked contract. This would later matter when the platform grew to 100 routes during the migration: the proof adapted to the real registry instead of preserving a stale magic number.
Codex also rejected a seductive shortcut: “it resolves in the monorepo” does not prove “it exists in the deployment ZIP.” Each production service needed its own frozen dependency graph and an exact-package boot test.
Phase 1: build a host-neutral API without rewriting a hundred handlers
Rewriting every handler into Express would have created unnecessary business-logic risk. Codex instead introduced a shared route-registration layer and a request/response adapter. Existing handlers could run behind both the Function host and the new Express server during the proving period.
The adapter had to preserve more than JSON. It handled route parameters, queries, headers, trusted absolute URLs, JSON, text, binary data, native multipart files, explicit response semantics, aborts, request-size limits and structured telemetry. An unknown backend path ended in a JSON 404 so it could never fall through to the web SPA.
The new runtime added lightweight /healthz and strict /readyz endpoints. Readiness reported host kind, Node version, deployment SHA, expected versus registered routes, configuration completeness, credential mode and bounded memory facts. It was a release contract, not a decorative “OK” endpoint.
The exact isolated Node 24 package booted successfully. The old Function package also remained provable during the transition. That dual-runtime stage was temporary scaffolding, not a promise of permanent backward compatibility.
Phase 2: provision the candidate, then distrust it professionally
Codex provisioned a small dedicated Linux service in the existing production subscription and resource group. It mirrored the already-understood shape of the separate AI service: fixed low-cost plan, Node 24 LTS, Always On, HTTPS-only, platform Health Check, restricted ingress and its own deployment credential. No new subscription was created.
Provisioning exposed schema drift in the cloud CLI: Linux plan and app-plan fields appeared under different property names than older scripts expected, and shell quoting could mutate the literal Node runtime string. The script was hardened to accept the observed schemas while remaining locked to the intended resource set.
Then came the more important part: proving the candidate in production without moving user traffic.
The proof ladder
- Assemble the exact production package from its frozen lockfile.
- Audit only the dependencies that will actually ship.
- Boot that package under Node 24 and probe it locally.
- Deploy the candidate behind default-deny ingress.
- Reach it from the allow-listed web worker.
- Require exact deployment SHA and full route-registry parity.
- Exercise real database and object-storage write/read/cleanup paths.
- Run maximum-shape, deliberately non-writing capacity probes.
- Retain sanitized evidence artifacts.
That ladder caught issues that an ordinary unit-test-and-deploy process would have missed.
Obstacle 1: a synthetic capacity proof was not the deployed service
An early local harness accepted a 50 MiB multipart file and measured peak memory. It looked useful, but the independent reviewer correctly challenged it: the load ran against synthetic handlers in another process. Codex removed the false-confidence harness and replaced it with a client executing from the deployed web worker against the real candidate worker.
The accepted proof sent a 52,428,800-byte multipart body and a 54,690,035-byte, 15,000-row JSON shape to deliberate pre-write rejection points while measuring concurrent health latency and post-request RSS. The point was not to brag about payload size. It was to characterize the single-instance B1 risk honestly.
Obstacle 2: the management container was not the application worker
A proof launched through the service-management environment could see deployed files but not the runtime worker’s mounted dependencies. That path was rejected. The proof moved into a temporary, heavily gated endpoint inside the real Node 24 worker: feature flag, trusted proxy credential, named admin principal, exact confirmation phrase and a single-flight lock.
Before cutover, Codex mechanically removed the temporary mutation route and required it to return 404. This converted a prose promise into a release gate.
Obstacle 3: the error response was too safe to diagnose
The first real worker failure returned only a generic sanitized error. Security was preserved, but operability was not. The reviewer pointed back to a previous production incident: admin-only maintenance endpoints could expose bounded error names and messages without leaking secrets. Codex added stage-specific error codes and log tails so the next failure could be investigated without another blind deploy cycle.
Obstacle 4: missing dependencies existed locally but not in the web artifact
The one-time provisioning proof imported a cloud storage SDK successfully in the repository but failed in the deployed web package. Codex converged on a proof-only bundle built from the frozen web lock and excluded from runtime dependency graphs. CI drift protection ensured the bundle could not silently age away from its lockfile.
Obstacle 5: a non-critical billing API was blocking a critical traffic gate
Every database, storage and route check was green, but a cost-management query was intermittently throttled. Treating it as a hard release failure would train operators to rerun until lucky. Codex reclassified billing visibility as a loud non-blocking advisory while keeping API, database, storage, route and capacity checks hard-fail. The retained artifact and System Health UI still disclosed the warning.
Obstacle 6: cleanup confirmation used a different identity
The canary wrote and deleted temporary storage markers with a connection-string client, then tried to confirm deletion by constructing a second client through a default credential chain. On the web worker, that second identity could not authenticate. The fix was not another role grant. It was to thread the already-authenticated client through cleanup confirmation. The final deployed proof created, read and cleaned all canary artifacts and confirmed absence using the same credential boundary.
Phase 2 ended with a retained same-SHA artifact proving the real worker, real data plane, route registry, health and characterized capacity. Only then was the candidate eligible for traffic.
Phase 3: one origin switch, one automatic rollback
The production cutover was deliberately narrow. The web proxy received one required backend-origin setting. There was no silent fallback. A hard-locked script verified subscription, resource names, runtime, shared trust, retired-canary absence and exact application SHA before changing that single value.
If the new backend failed its authenticated proof, the script restored the Function origin and proved the rollback before returning failure.
The first switch rolled back—and that was success
The first cutover appeared green from the control plane and direct backend probe, but the signed-in System Health page could not fetch the API. Codex rolled back immediately. Investigation showed that a deployment marker could update before the active web worker had actually restarted onto the same release.
The workflow was changed to wait for public web health to report the exact worker SHA, not merely for a file in the management container. A later same-SHA release passed, traffic moved, and System Health identified the dedicated App Service, Node 24, full route parity, zero timers, working storage and database probes, healthy AI service and embedded ETL probes.
This was an important cultural moment: rollback was not treated as embarrassment. It proved the safety system worked before customer impact.
The weekly ETL was the real business acceptance test
API migrations are not accepted by health endpoints alone. The platform’s weekly ERP workflow uploads multiple operational exports, runs extraction and synchronization, publishes manifests, invalidates caches and renders downstream dashboards.
I started the real weekly uploads through the Admin console while Codex monitored the platform. The complete cycle succeeded through the Node 24 backend. Sales, expenses, debtors, order book, delivery, stock and purchase-order data refreshed; downstream dashboards rendered the current fiscal-year views and week-over-week movements.
When the purchase-order ETL showed a large anomaly increase, Codex did not contaminate the migration scope. At my direction, that data-quality investigation was forked into a separate clean task. This sounds procedural, but it is executive discipline: do not let every production observation become an excuse to destabilize an otherwise controlled migration.
Cost-frugal telemetry for a single small server
Serverless had hidden CPU and memory planning. A dedicated single-instance API made them my responsibility. I wanted enough observability to make an evidence-based decision without buying an enterprise APM stack for a small internal application.
Codex built a zero-additional-cost approach from existing platform metrics, bounded operational logs and the Admin System Health page. Seven email alerts covered CPU, application working set, plan memory, request queue, clustered 5xx responses, Health Check availability and response time. A daily read-only capture compared 24-hour aggregates and looked specifically for monotonic working-set growth.
| Soak observation | Measured result | Interpretation |
|---|---|---|
| 24-hour request volume | 2,147 requests | Representative small-platform production traffic. |
| CPU average / p95 / max | 9.7% / 13.6% / 35% | Comfortable headroom on the initial B1 plan. |
| Application working set avg / p95 / max | 144 / 240 / 282 MiB | No evidence of memory saturation or monotonic growth. |
| HTTP queue | 0 | No observed request backlog. |
| Health Check | 100% | The worker stayed available throughout the sample. |
| Platform response aggregate avg / p95 / max | 36 ms / 164 ms / 1.68 s | Good platform latency; explicitly not mislabelled as per-request percentile telemetry. |
| HTTP 5xx | 1 of 2,147 (0.0466%) | No corresponding handler failure event, queue, health or memory signal; classified as a one-off unattributed edge failure. |
| Later daily review | 0 HTTP 5xx; queue 0; health 100%; working-set p95 about 210 MiB | The earlier 5xx did not form a pattern. |
Codex also audited the code for leak risks: timers, in-flight maps, clients, caches, listeners, streams and timeouts. No confirmed leak was found. It recorded two honest watch items instead: a count-bounded cache was not yet byte-bounded, and very large JSON or multipart bodies necessarily create transient RSS pressure. The evidence did not justify a speculative cache rewrite during soak.
Phase 4: stop, do not delete
After the weekly ETL and a later financial-pack ETL succeeded, I became bullish about completing the migration. I no longer wanted the Function API treated as a supported cold spare, but I also did not want storage, identity, networking or other cloud configuration disturbed. The least-blast-radius answer was precise: stop the Function App and change nothing else.
Codex created an annotated release checkpoint, verified the tag’s peeled commit, required green production workflows at the exact SHA, checked the active web origin, and reached the dedicated API from the web worker. The deactivation script fingerprinted the legacy resource’s settings and restrictions before the stop and compared them after.
The first attempt failed closed before touching Azure because Windows PowerShell retained an unrelated native-process exit code across a piped Git query. Codex corrected the guard, required a new exact-SHA release and a suffixed tag, then reran the process.
The only production mutation was the stop command. Post-action verification confirmed the legacy resource was stopped; its plan, storage, content, identity, settings and network restrictions remained present; the dedicated API stayed healthy and authoritative; and no credentials, role assignments or unrelated platform resources changed.
The retained Function can be used one day for a different serverless feature, but only through a new workstream. Starting it does not resurrect a valid business API.
Phase 4A: de-Functionize the source code, not the cloud account
A stopped resource was not enough. The repository still contained Function-shaped names, adapters, registration metadata, local configuration and deployment concepts. Future coding agents could interpret those as a compatibility requirement and keep adding features to a dead path.
Before the surgical cleanup, I asked Codex to refresh against a newly merged material-cost feature branch. It froze the now-current surface at 100 routes and preserved the external-publishing scheduler that belonged to the Node process rather than the retired Function host.
Codex renamed the authoritative route tree, replaced the Function registration facade with an HTTP route registry, replaced the Function-shaped adapter with an Express adapter, removed Function-only metadata and a dormant timer, renamed active local configuration, removed runtime fallback to Function-named storage settings, added a permanent anti-regression contract, and reconciled the complete documentation surface.
The complete repository gate passed 408 test files, 2,813 tests, a 40/40 authorization golden matrix, a 160/160 pairwise matrix, 545 AI-core tests, the production build, a zero-high-vulnerability frozen dependency audit and a secret scan across 1,678 tracked files. The final package booted under Node 24 with 100/100 route parity.
The local stack mattered. A fresh Node 24 setup with database and object-storage emulators, Express and Vite initially stopped at Loading…. Codex traced the failure to generated local configuration that lacked a required cache-encryption secret. The generator received a fixed emulator-only value; production remained independently secret-managed. After restart, the authenticated proxy and representative business routes returned 200 through the real local architecture.
This is what “surgical” meant: remove obsolete concepts, preserve current behavior, prove the full route tuple, and leave the unrelated cloud resource untouched.
The final obstacle was not code—it was GitHub Actions itself
The final cross-runtime release should have produced three workflows: web and embedded ETL, platform API, and AI service. During a GitHub Actions incident, the AI track completed, while the web and API runs appeared queued with zero jobs.
After GitHub reported recovery, those two records still did not move. Codex checked workflow activation, repository permissions, concurrency, billing, job-level state and the official status API. The evidence was contradictory: the records said queued, cancellation said they had not been queued yet, and rerun said they were already running. They were outage-orphaned control-plane objects, not active jobs.
Codex created a temporary branch pointing to the exact reviewed application commit and dispatched only the missing web and API workflows from that ref. Jobs materialized immediately. All three tracks passed build, test, audit, package, deployment and smoke gates at the same SHA. Live web health and API readiness returned that exact commit; the API reported Node 24 and 100/100 routes. The temporary branch was then deleted.
The stale zero-job records remain harmless UI residue because GitHub will not cancel an object its scheduler never accepted. Codex did not delete historical workflow evidence or trigger duplicate production races merely to make the page look tidy.
The numbers behind the migration
Other useful measurements: 2,147 requests in the representative day, zero queue, CPU p95 13.6%, working-set p95 about 240 MiB, one unattributed edge 5xx that did not recur, zero high-severity vulnerabilities in the frozen API dependency graph, and one successful weekly ERP cycle plus a later financial-pack ETL before the legacy host was stopped.
The token economics of a persistent autonomous agent
This was not a ten-message chat. The Codex task remained alive from August 19 through August 26, carrying plans, repository contracts, tool output, production evidence, reviewer feedback and owner steering across many context compactions.
| Session counter before this blog request | Measured value | How to interpret it |
|---|---|---|
| Total processed tokens | 357,737,970 | Input plus output processed by the long-lived task; not 358 million freshly typed words. |
| Input tokens | 356,809,023 | Includes repository context, tool results, accumulated conversation and replayed state. |
| Cached input tokens | 348,989,184 (97.81%) | Most context throughput reused cached input, which is what makes persistent agentic work more economically plausible. |
| Uncached input tokens | 7,819,839 | The more meaningful measure of newly processed input within the available counter. |
| Output tokens | 928,947 | Agent responses and execution-oriented output across the task. |
| Reasoning tokens | 229,780 | A reported subset of model output used for internal reasoning. |
| Tool calls | 2,593 | Repository reads, tests, scripts, cloud queries, deployments, browser checks, monitoring and task orchestration. |
| User-channel inputs | 58 | Owner directions plus scheduled heartbeat prompts; not 58 low-level implementation instructions. |
These are raw Codex task counters captured immediately before writing this post. They are workload telemetry, not a billing statement. The total is dominated by cached context and repeated tool-rich state, so it should not be compared directly with a conventional one-shot chat or used as a simple cost multiplier.
The important ratio is not “tokens per line of code.” It is decision leverage. Fifty-eight user-channel inputs guided more than two thousand tool actions across code, CI/CD, cloud infrastructure, browser testing and monitoring, while the same task retained the architectural history needed to avoid rediscovering every decision.
What Codex autonomy looked like in practice
Autonomous execution was not one giant prompt. It was a control loop:
read the governing contracts
→ inspect the real repository and live state
→ propose the smallest reversible work package
→ implement and test both sides of each interface
→ deploy the exact artifact
→ collect observable evidence
→ invite independent challenge
→ correct or roll back
→ update the durable ledger
→ ask the human only for decisions that change authorityCodex did not merely suggest cloud commands. It ran the safe ones. It did not merely propose tests. It built the exact package and executed the gates. It did not merely say “monitor memory.” It provisioned cost-free alerts, captured baselines, inspected the 5xx and audited code for retained state. It did not merely recommend a rollback. It encoded one and used it when the first cutover proof failed.
The agent also absorbed steering without losing the thread:
- When I said the external quote-system API was a later workstream, it kept that integration separate.
- When I asked for cost frugality, it used existing metrics rather than paid telemetry.
- When I became bullish after live ETLs, it still chose “stop, do not delete” as the smallest blast radius.
- When I insisted the Function resource might have a different future, it removed business-API coupling without destroying the cloud asset.
- When a major material-cost feature merged, it refreshed the route baseline before refactoring.
- When GitHub itself failed, it diagnosed scheduler state before redispatching only what was missing.
Where the independent reviewer earned its place
I asked Codex to bring in an independent review agent because autonomy without adversarial review can become confidence theatre. Claude’s feedback repeatedly improved the work:
- It identified the workflow race between web deployment and API same-SHA verification.
- It rejected generic canary errors that made production failures undiagnosable.
- It caught the deployment-artifact dependency that local tests concealed.
- It insisted the write/read/cleanup proof run in the deployed context.
- It challenged hard-fail treatment of a throttled, non-serving billing API.
- It traced cleanup confirmation to a differently authenticated client and argued against unnecessary role grants.
- It reviewed the final de-Functionization boundary and route parity before giving a GO verdict.
Codex did not accept every suggestion blindly. It checked each one against code, logs and current state. That builder-reviewer tension, with the owner deciding business and risk policy, was healthier than either a single-model monologue or a human copying opinions between chat windows.
Ten lessons I would carry into another enterprise migration
- Read incident-derived contracts before touching code. The fastest route is the one that does not repeat last quarter’s failure.
- Freeze observable behavior before changing host technology. Route manifests, response parity and authorization matrices are migration assets.
- Test the artifact, not the repository. Monorepo dependency success is not deployment-package success.
- Deployed proof outranks local proof. Especially for credentials, mounts, networking, storage and write paths.
- Thread authenticated clients; do not re-derive identity. A second credential path can invalidate an otherwise correct canary.
- Separate serving health from advisory telemetry. A throttled cost API should be visible, not able to veto customer traffic.
- Make rollback executable and observable. The first rollback was evidence of maturity, not failure.
- Use the human at irreversible boundaries. Architecture, money, traffic, deletion and external activation remain owner decisions.
- Remove obsolete mental models from the codebase. Dead compatibility code is an instruction to future engineers and agents.
- Document final state twice. Once when the new service becomes authoritative, and again when retained legacy infrastructure changes status.
The leadership point
The most interesting outcome is not that an AI agent can write an Express adapter or a cloud script. The important outcome is that one person can direct a production-scale maintenance programme with the discipline normally associated with a much larger platform team.
That requires more than prompting skill. It requires architecture judgment, cost governance, separation of concerns, the confidence to authorize progress, the restraint to preserve optionality, and the operational habit of demanding evidence. The agent amplified those qualities; it did not replace them.
My definition of successful AI engineering is not “the model wrote a lot of code.” It is this: the human can explain every important trade-off, the agent can execute the agreed path with persistence, the reviewer can challenge the evidence, production can prove the result, and the next engineer encounters a clearer system than the one we started with.
The retirement email asked for Node 24. We delivered something more valuable: a coherent platform backend, a repeatable autonomous-engineering method and a codebase with one unambiguous future.
Publication note: all cloud names, endpoints, account identifiers, credentials, principals and proprietary business data have been redacted or generalised. Measurements are retained where they communicate engineering scale without exposing business content.
No comments:
Post a Comment