Server Ops on UIPad Blog

Docker + Postgres Auth Voodoo: A Troubleshooting Log of a Fatal Nano Auto-Wrap Bug

Sun, 26 Apr 2026 21:30:00 +0800

As an indie developer who constantly tinkers with servers, dealing with Docker is a daily routine. I thought deploying a PostgreSQL instance via docker-compose up -d was a standard operation I could do with my eyes closed. However, I recently fell headfirst into a cascading series of traps.

The error message is one we are all too familiar with:

FATAL: password authentication failed for user "app"

But after repeatedly verifying that my environment variables, network configurations, and even inter-container connectivity were all flawless, things started getting weird. After two days of peeling back the layers, I discovered this wasn’t just a simple password typo. It was a triple-threat trap composed of phantom configurations, volume mounting blind spots, and editor quirks.

This post logs the entire debugging process, hoping to save the sanity of anyone else doubting their existence in the terminal.

Trap 1: The “Two Negatives Make a Positive” Phantom Password

The whole thing started like this: I noticed my newly created App container absolutely refused to connect to Postgres, yet another Auth service I had deployed earlier was connecting just fine.

At first, I wondered if routing through the internal Docker network (e.g., Host=pgsql) bypassed the password check. But Postgres’s pg_hba.conf mechanism dictates that TCP connections must use scram-sha-256 validation; passwordless entry simply isn’t an option here.

By directly running an echo on the environment variables inside the container, I caught the first mole: unquoted .env comments.

In an earlier iteration of the project, I had an .env file written like this: PGSQL_APP_PASSWORD=app@pgsql # Global app password

And my docker-compose.yaml looked like this:

POSTGRES_APP_PASSWORD: ${PGSQL_APP_PASSWORD:-app@pgsql}

Because the .env value wasn’t enclosed in double quotes, Docker Compose brutally swallowed the trailing spaces and the inline comment right into the environment variable.

During database initialization, it saved app@pgsql # Global app password as the complete, literal password.
The old Auth container read that exact same configuration and made requests using that ridiculously long, comment-included password. Both sides matched perfectly (two negatives made a positive)!
Meanwhile, when I manually typed the clean password app@pgsql in the command line, I was ruthlessly rejected.

Pro Tip: For complex strings in .env files or docker-compose.yml, especially passwords with special characters, always make it a habit to use double quotes.

Having discovered the issue above, I decided to fix the config, switch to a clean password app#pgsql, and restart the container. The result? Still throwing the same error!

I went inside the container, printed the environment variable, and confirmed $POSTGRES_APP_PASSWORD was now the correct app#pgsql. Why couldn’t it connect?

That’s when I noticed this line in my docker-compose.yml:

volumes:
 - ./data:/var/lib/postgresql/data

This triggers a classic “hidden rule” of the Postgres Docker image: As long as the mounted target directory (/var/lib/postgresql/data) is not empty, the container startup will completely skip the initialization process (including the execution of scripts under /docker-entrypoint-initdb.d/).

Because I had already spun it up once, the ./data directory on my host machine contained legacy data. Even though I changed the config and restarted the container, the password stored inside the database was still that Chinese-comment-infused “phantom password” from the first initialization.

Pro Tip: When debugging initialization scripts in a development environment, if you change passwords or the initial database structure, you must wipe the host’s ./data directory (rm -rf ./data/*), or drop into the container and forcefully update the password using ALTER USER.

Final Boss: The “Backstab” of Nano’s Copy-Paste

Alright, I deleted the old data and was certain it would re-initialize this time. The docker logs clearly showed my custom script 01-init-user-and-permissions.sh executing, even printing NOTICE: User created: app.

I eagerly went to test the connection. Error: password authentication failed.

At that moment, I genuinely started to question if my fundamental understanding of Docker had been tampered with. The network was fine, the environment variables were fine, the data had been wiped clean—where was the ghost?

It wasn’t until I opened that 01-init-user-and-permissions.sh script and scanned it line by line that I finally spotted the incredibly stealthy fatal flaw:

CREATE USER "$POSTGRES_APP_USER" WITH PASSWORD '$POSTGRES_APP_PASSWO
RD');

Yes, you read that right. The variable name $POSTGRES_APP_PASSWORD had been chopped in half by a line break!

How did this happen? Because I was working directly in an SSH terminal, using the nano editor, and I copied and pasted the script from another server. When your terminal window isn’t wide enough, and nano has its default Word Wrap feature enabled, it takes the liberty of inserting an actual Hard Return right in the middle of long strings.

In Bash logic, it couldn’t find the variable $POSTGRES_APP_PASSWO (because it was truncated by the return key), so it parsed it as an empty string. Ultimately, the database successfully executed the SQL, but what it actually ran was: CREATE USER "app" WITH PASSWORD '';

The password was blank! This was the ultimate reason why my perfectly correct password was constantly failing.

Summary & Best Practices

Stacking these three pitfalls together creates maximum dramatic effect. To prevent myself (or you, the reader) from falling into this trap again, here are a few ironclad rules for development:

Terminal Editor Self-Defense: If you need to paste long code or configs into nano on a Linux terminal, always remember to add the -w flag:
```
nano -w script.sh
```
This disables auto word wrapping and is an absolute lifesaver. An even better approach is using VS Code’s Remote SSH extension to edit server files directly, bypassing terminal clipboard torture entirely.
Environment Variable & Password Standards: Wrap .env passwords and variables containing special characters in double quotes. Keep inline comments on their own separate lines.
```
# Global app password
PGSQL_APP_PASSWORD="app@pgsql"
```

The 3-Step DB Debugging Axis: When encountering Docker DB password voodoo, drop straight into the host with superadmin privileges to investigate and settle it instantly:

# 1. Check what configuration the container actually swallowed
docker exec -it pgsql /bin/bash
echo $POSTGRES_APP_PASSWORD

# 2. Force a connection test using the exact environment variable
PGPASSWORD="$POSTGRES_APP_PASSWORD" psql -h 127.0.0.1 -U app -d postgres

Server operations are just like this. Sometimes, what tortures you for two straight days isn’t some low-level kernel bug, but simply an invisible line break. Documenting this down, mostly as a stark reminder to my future self!

Cross-Cloud Database Resilience: PostgreSQL Streaming Replication with Dual-Cert SSL

Sat, 14 Mar 2026 21:00:00 +0800

As an indie developer, “Single Point of Failure” is the stuff of nightmares. While Oracle Cloud offers a generous Free Tier, relying on a single provider for your production data is risky. To ensure data sovereignty and high availability, I recently implemented a cross-cloud PostgreSQL streaming replication setup.

This isn’t just a simple cron-job for backups—it’s a real-time “Shadow Database” architecture.

1. The Need for Real-Time Replication

Relying solely on daily snapshots had two major drawbacks:

RPO (Recovery Point Objective): A crash at 11 PM meant losing a whole day of data if the last backup was at 2 AM.
RTO (Recovery Time Objective): Restoring Giga-bytes of data from S3 to a new container during an outage takes far too long.

My Goal: Maintain a read-only replica on a separate VPS, ready for a minute-level failover.

2. The Solution: mTLS + Streaming Replication

Since data travels over the public internet, security was non-negotiable.

Encryption: TLS is mandatory to prevent sniffing.
Authentication: Instead of passwords, I used Mutual TLS (mTLS). Only the server holding a client certificate signed by my private CA can connect to the primary.
Isolation: Custom ports (e.g., 8765) combined with strict IP whitelisting via iptables.

3. Hard-Won Lessons (The Pitfalls)

Pitfall #1: Docker vs. System Firewall

Docker often bypasses standard ufw rules by manipulating iptables directly.

Fix: Explicitly insert rules into the INPUT chain for the specific standby IP and persist them using iptables-persistent.

Pitfall #2: The pg_basebackup Path Ghost

When running pg_basebackup in a temporary container, it hardcodes the ephemeral paths (like /temp_certs) into postgresql.auto.conf.

Fix: Manually edit the config post-clone to point to the permanent volume paths (e.g., /var/lib/postgresql/data/certs/).

Pitfall #3: Permissions (The 999 Rule)

PostgreSQL is notoriously picky about certificate permissions.

Error: could not open file "server.key": Permission denied.
Fix: Run chown -R 999:999 on the host data directory. Even if it looks like root on the host, the container process must own it as UID 999.

4. Key Configurations

Primary `pg_hba.conf`

# Only allow replication_user via certificate-based auth
hostssl replication replication_user <Standby_IP>/32 cert

Standby `postgresql.auto.conf`

primary_conninfo = 'user=replication_user host=<Primary_IP> port=8765 sslmode=verify-ca sslcert=/path/to/client.crt sslkey=/path/to/client.key sslrootcert=/path/to/ca.crt'

5. Maintenance: Why I Switched to Bind Mounts

I moved away from Docker Named Volumes to Bind Mounts.

Reason: When migrating the standby to a new server, a simple tar of /opt/pgsql/data is transparent. No more hunting for obscure Docker volume hashes. It makes migration 200% faster.

Conclusion

High Availability isn’t about showing off; it’s about sleeping better at night. Currently, this setup maintains near-zero lag under Uptime Kuma monitoring.