When the Nix Eval Cache Serves You a Ghost Derivation

The project config said 15432. The process was on 15433. Tests could not connect, because they were dialing the port the app expected and nothing was answering there.

I opened devenv.nix. Port 15432. I opened .env. Port 15432. I grepped the whole repo for 15433. Nothing. Not a single reference. And yet devenv up had booted Postgres on 15433, and it had done it confidently, with no warning, every single time.

So the question was simple and infuriating. Neither .env nor devenv.nix mentions 15433 anywhere. Both say 15432. Where is Postgres getting 15433 from?

One caveat before blaming the cache: in devenv 2.x, the Postgres port is also the base for automatic port allocation, so 15433 can be legitimate if 15432 is already occupied. That did not make the mismatch less real. It just meant I needed to find the exact generated config the running process had loaded.

What is actually listening

First, confirm reality. The config is a claim; the running process is the fact. So I went and looked at the fact.

The Postgres that was actually serving had its data directory at .devenv/state/postgres, and that directory had a postgresql.conf in it. Line 17:

port = 15433

There it is. Not in any file I had edited. Sitting in the runtime state directory that devenv manages. So the next question writes itself. Who put 15433 in .devenv/state/postgres/postgresql.conf?

Walking the process tree

In this setup, devenv did not start Postgres directly. It started a process manager wrapper, which ran a chain of generated scripts. Following the running process back up:

devenv-processes-postgres
  -> /nix/store/vy3ndzy...-start-postgres/bin/start-postgres
    -> /nix/store/f7zr5mg...-setup-postgres/bin/setup-postgres

setup-postgres is the interesting one. It is the script that prepares $PGDATA before the server boots. And the way devenv ships a postgresql.conf is not by templating it into the state directory at runtime. It bakes the config into the Nix store at evaluation time and then copies it into place. The relevant line inside that setup script:

cp /nix/store/dasdnbj41...-postgresql.conf "$PGDATA/postgresql.conf"

So the postgresql.conf in my state directory was not authored by me and was not derived from the devenv.nix I had open. It was cp-ed out of a specific store path. Read that store path:

$ cat /nix/store/dasdnbj41rjsw5hnp94vw2vfdllqvz20-postgresql.conf
...
port = 15433

The store object had 15433 baked in. The lie was now located. It lived in an immutable Nix store path, and setup-postgres faithfully stamped it into my data directory every time it ran.

The part where it gets genuinely weird

Here is the catch. My current shell's PATH did not point at that derivation.

When I checked which start-postgres was on my PATH, it resolved to a different store hash:

/nix/store/90bj99hi3f1d1vffwl34qbag44f15043-start-postgres/bin/start-postgres

But the process actually running had been launched from:

/nix/store/vy3ndzyhaa961vd5swbahdlx0hzafi0w-start-postgres/bin/start-postgres

Two different start-postgres derivations. The one in my shell would copy a 15432 conf. The one that the live process had actually come from was an older build that copied the 15433 conf.

Same name. Different content. My environment was internally inconsistent: the interactive shell knew about the new world, but the running daemon was still a fossil from the old one. That alone did not prove the eval cache was guilty; it proved the live process was not using the derivation I thought it was using. That is why grepping the repo found nothing. The 15433 was never in the repo. It was frozen inside a store object that the live process was still pointed at.

Wait. If devenv.nix already says 15432, why was the old derivation still getting revived?

The eval cache is the part to distrust

devenv keeps an evaluation cache so it does not have to re-run Nix evaluation on every command. It lives here:

.devenv/nix-eval-cache.db
.devenv/nix-eval-cache.db-shm
.devenv/nix-eval-cache.db-wal

That cache maps evaluated attributes to the files, environment variables, and options they depended on. It is supposed to invalidate when a source file read during evaluation changes. Fast in the common case. A trap when that dependency tracking goes sideways.

The cache should have noticed my edit and rebuilt the process config. But the result I kept getting was the stale start-postgres derivation, and forcing the cache to refresh was what finally broke the loop. That old derivation referenced the stale postgresql.conf store path. setup-postgres copied 15433 into my data directory. Postgres booted on 15433. Every restart reproduced the exact same wrong port, which is precisely why it felt like gaslighting. A flaky bug you can dismiss. A bug that is perfectly consistent and contradicts the file in front of you is the one that makes you doubt your own eyes.

The fix

To get unblocked for tests immediately, the cheapest move is to stop arguing with the running process and just point the client at where Postgres actually is. Set the port in .env to match reality:

DB_PORT=15433

Postgres was already up on 15433, so tests connect right away. One line, reversible, buys you time.

But that is a workaround, not a fix. The actual repair is to force devenv to refresh its eval cache so it re-reads the file I edited:

# stop the devenv processes first
devenv up --refresh-eval-cache

If that still leaves the cache wedged, the sledgehammer version is deleting .devenv/nix-eval-cache.db and its SQLite sidecar files, if they exist.

With the cache refreshed, devenv re-evaluates devenv.nix, produces a fresh start-postgres derivation, that derivation references a postgresql.conf store path with port = 15432, and setup-postgres copies the correct conf into $PGDATA. Postgres comes up on 15432. The config and the process finally agree, and I reverted the .env line.

There is a tempting third option I deliberately avoided: hand-editing .devenv/state/postgres/postgresql.conf from 15433 back to 15432. It works until the next devenv up, at which point setup-postgres cheerfully cps the stale store conf right back over your edit. Editing the copy is pointless when the source of the copy is the thing that is wrong.

The Nix lesson is the one worth carrying out of this. There are three layers and you have to keep them straight: the config you declare in devenv.nix, the immutable artifact that config evaluates to in the store, and the eval cache that is supposed to decide when to re-derive that artifact. When the running process diverges from the file you are editing, the file is rarely the liar. Walk down to the store path the process actually loaded, account for port allocation, and if that path is older than your last edit, suspect the process state and the cache that handed it to you.