Csdrhrt

I have a curious problem - very similar to the question here, but none of the answers have worked for me.

The problem

Public key authentication fails from the primary to the replica node. I used ssh-keygen to create a new key under the "postgres" user on each machine, and then used ssh-copy-id to copy each machine's public key to the other.

Under the postgres account, this only works from the replica to the primary, not vice-versa. Additionally, this process works both ways for all other users on the machines.

Files and logs:

First, the ssh log (truncated slightly):

ssh -vvv 192.168.1.104

OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 58: Applying options for *

debug2: resolving "192.168.1.104" port 22

debug2: ssh_connect_direct: needpriv 0

debug1: Connecting to 192.168.1.104 [192.168.1.104] port 22.

debug1: Connection established.

debug1: identity file /var/lib/pgsql/.ssh/id_rsa type 1

debug1: key_load_public: No such file or directory

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_7.4

debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4

debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000

debug2: fd 3 setting O_NONBLOCK

debug1: Authenticating to 192.168.1.104:22 as 'postgres'

debug3: hostkeys_foreach: reading file "/var/lib/pgsql/.ssh/known_hosts"

debug3: record_hostkey: found key type ECDSA in file /var/lib/pgsql/.ssh/known_hosts:1

debug3: load_hostkeys: loaded 1 keys from 192.168.1.104

debug2: compression ctos: none,zlib@openssh.com

debug2: compression stoc: none,zlib@openssh.com

debug2: languages ctos: 

debug2: languages stoc: 

debug2: first_kex_follows 0 

debug2: reserved 0 

debug1: kex: algorithm: curve25519-sha256

debug1: kex: host key algorithm: ecdsa-sha2-nistp256

debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

debug1: kex: curve25519-sha256 need=64 dh_need=64

debug1: kex: curve25519-sha256 need=64 dh_need=64

debug3: send packet: type 30

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

debug3: receive packet: type 31

debug1: Server host key: ecdsa-sha2-nistp256 SHA256:D8LwBEs1BC7iSlmZOa3b58/gS6mRIIxfGHl6buMJA/c

debug3: hostkeys_foreach: reading file "/var/lib/pgsql/.ssh/known_hosts"

debug3: record_hostkey: found key type ECDSA in file /var/lib/pgsql/.ssh/known_hosts:1

debug3: load_hostkeys: loaded 1 keys from 192.168.1.104

debug1: Host '192.168.1.104' is known and matches the ECDSA host key.

debug1: Found key in /var/lib/pgsql/.ssh/known_hosts:1

debug3: send packet: type 21

debug2: set_newkeys: mode 1

debug1: rekey after 134217728 blocks

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug3: receive packet: type 21

debug1: SSH2_MSG_NEWKEYS received

debug2: set_newkeys: mode 0

debug1: rekey after 134217728 blocks

debug2: key: /var/lib/pgsql/.ssh/id_rsa (0x556e0db5a510)

debug2: key: /var/lib/pgsql/.ssh/id_dsa ((nil))

debug2: key: /var/lib/pgsql/.ssh/id_ecdsa ((nil))

debug2: key: /var/lib/pgsql/.ssh/id_ed25519 ((nil))

debug3: send packet: type 5

debug3: receive packet: type 7

debug1: SSH2_MSG_EXT_INFO received

debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>

debug3: receive packet: type 6

debug2: service_accept: ssh-userauth

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug3: send packet: type 50

debug3: receive packet: type 51

debug1: Authentications that can continue: publickey

debug3: start over, passed a different list publickey

debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password

debug3: authmethod_lookup publickey

debug3: remaining preferred: keyboard-interactive,password

debug3: authmethod_is_enabled publickey

debug1: Next authentication method: publickey

debug1: Offering RSA public key: /var/lib/pgsql/.ssh/id_rsa

debug3: send_pubkey_test

debug3: send packet: type 50

debug2: we sent a publickey packet, wait for reply

debug3: receive packet: type 51

debug1: Authentications that can continue: publickey

debug1: Trying private key: /var/lib/pgsql/.ssh/id_dsa

debug3: no such identity: /var/lib/pgsql/.ssh/id_dsa: No such file or directory

debug1: Trying private key: /var/lib/pgsql/.ssh/id_ecdsa

debug3: no such identity: /var/lib/pgsql/.ssh/id_ecdsa: No such file or directory

debug1: Trying private key: /var/lib/pgsql/.ssh/id_ed25519

debug3: no such identity: /var/lib/pgsql/.ssh/id_ed25519: No such file or directory

debug2: we did not send a packet, disable method

debug1: No more authentication methods to try.

Permission denied (publickey).

I'm absolutely stumped. Are there other reasons that the public key authentication might fail to work? What troubleshooting should I perform from here?

Edit

Disabling SELinux has fixed the problem, but leaves me with a number questions - some of which are auxiliary and perhaps out of scope for this, but that I think may be useful to future visitors nonetheless:

1. How do I solve this problem without disabling SELinux? I failed at using audit2allow, Can this be accomplished without any third-party tools (i.e., only what is available on the CentOS 7 minimal ISO)?


2. Why would SELinux policies be different between the two virtual machines? They were both cloned from the same template.


3. Why would SELinux only affect one user account, but not another?