IfModule doesn't work

So in the interest of smaller apache conf files, I separated all of my VHosts into their own files. Everything works fine except CBand. For some reason this directive doesn’t work:

<IfModule mod_cband.c>
CBandSpeed 1024 10 30

When the VirtualHost directives were in httpd.conf it worked fine. Is there a way to make this work? Or will I have to put them all back into httpd.conf?

Thank you for your time