Pretty sure it’s already allowed and already doing what you’ve asked for. Postfix definitely does not require encryption on the traditional SMTP port, it uses opportunistic encryption (if the client can support it, it’ll enable it).
Looks like maybe Dovecot needs a minor tweak to allow pop3 and imap instead of just pop3s and imaps. On CentOS, that’s probably changing /etc/dovecot/conf.d/10-ssl.conf to have ssl = yes rather than ssl = required.
Removing the certificate will just break all connections, it will not allow unencrypted connections. The existence of a certificate doesn’t configure the services for TLS (or not).
So, you could do that, but you definitely should not, as it will not do anything good or useful to your goals.