You shouldn’t have an AAAA record if you don’t have a VirtualHost to match it. So, either delete the AAAA record, or configure a VirtualHost on the IPv6 address. You can’t have it both ways. Either offer IPv6 or don’t, but don’t pretend someone can find your website on an IPv6 address when you aren’t actually serving one.
DNS is a caching protocol. If it existed in the past, it can take up to the TTL to disappear from cached copies of that record (longer for DNS servers that ignore TTL, but I doubt Let’s Encrypt uses broken DNS servers).