Ghost Web App Installer Upgrade Removes Proxy

SYSTEM INFORMATION
OS type and version Rocky Linux 9.7
Virtualmin version 8.0.0 Professional

I have a couple Ghost websites on my server. Each has been installed via your Web Apps functionality and has been running for around two years. Each has Ghost running in the main root web directory rather than a subdirectory. I regularly use the Ghost Update functionality in Virtualmin’s Web App Installers section of the System Settings.

Yesterday I clicked the button on one of my Ghost sites to upgrade from Ghost 6.13.2 to Ghost 6.15.0. Usually clicking this button runs without a hitch. No errors were reported, however when I visited the website to verify the upgrade, I was greeted by a 403 Forbidden error.

My first thought was to revert the upgrade, which I did on the command line by stopping Ghost (after first checking everything with Ghost Doctor) and then running:

ghost update 6.13.2 --force --no-setup-linux-user

The installer confirmed all was well and up-and-running. So I was surprised to be greeted again by the 403 Forbidden error, even though no errors were reported on the command line. I tried updating to 6.15.0 again, and of course received the same results.

I poked around Apache’s logs (since there were no new Ghost error logs), and compared this Ghost site that wasn’t working to the other working Ghost site that I hadn’t yet upgraded. I found the problem, but have yet to be able fully to fix it!

When I looked under the domain’s Web Configuration menu and selected both Configure Apache Website and also Configure Apache SSL Website, there was a missing entry in the Per-Directory Options, which should have been the following (after changing the Port to the one for this install):

Proxy balancer://mongrel3000

I added that entry to both SSL and non-SSL configuration.

I also went in and added the appropriate Directive:

BalancerMember http://localhost:3000

I also noted that under Aliases & Redirects, there were related missing entries that I added:

balancer://mongrel3000

I restarted Apache for all my changes to take effect. No errors doing so, happily! I did all of the above without knowing quite what I was doing, other than mirroring the working Ghost installation that I had not yet upgraded.

And now my Ghost homepage did start showing up, but only with its text content (so with none of its connected assets like styles and images). When I tried to visit the Ghost Dashboard, I now got an Internal Server Error rather than a 403 Forbidden error.

I tried every which way to figure out how to fix this myself! I logged in as root to my server and poked around with listing proxies there. I noticed something amiss when I ran this command:

sudo virtualmin list-proxies --domain redacted.com --multiline

/.well-known
    Proxying: No
    Websockets: No
/
    Balancer: mongrel3000
    Proxying: Yes
    Script name: ghost
    Script version: 6.15.0
    Websockets: No

There’s a missing URL entry above, but when I tried to add it with this command, it still didn’t show up, even though I received no error after issuing this command:

sudo virtualmin modify-proxy --domain redacted.com --path / --url http://localhost:3000

I tried this using the GUI too, and even when I add that URL and save, it disappears without error :frowning:

There was one more thing that I checked, inside the httpd.conf file for this website, there were missing entries which I added:

<Proxy "balancer://mongrel3000">
    BalancerMember http://localhost:3000
</Proxy>
ProxyPass / balancer://mongrel3000
ProxyPassReverse / balancer://mongrel3000

The only entry that wasn’t missing was this one: ProxyPass /.well-known !

How on earth did all these entries disappear simply when I clicked the upgrade button for Ghost from 6.13.2 to 6.15.0? The upgrade button has never done this before, and again, no errors were reported, so I expected to simply find an updated Ghost install as usual.

As a last-ditch attempt to see what was going on, I created a fresh Ghost install using the Web App Installer into the root directory of a domain I’m not actively using but that was all setup in Virtualmin. Ghost installed just fine, but I noticed something particularly about this new install’s proxy settings. Instead of mongrel it’s now proxy as in:

Proxy balancer://proxy3001

I had to throw in the towel for the night at this point, being over my head in terms of knowledge of proxying Node servers with Apache. I would love to know how to resurrect my website, which is currently still not working since deciding to update it last night! Ghost reports it is running just fine on port 3000, but somewhere I’ve missed adding or fixing a proxy entry, hence the broken homepage (errors fetching any linked assets) and the Internal Server Errors when trying to access Ghost’s Dashboard URL.

1 Like

Yes, this is all true—it’s a bug.

This is expected and isn’t relevant to the problem, because directives are removed during an upgrade either way.

@Jamie, I see it too. Is it related to your latest changes? Check the diff produced by the Ghost upgrade:

diff --git a/httpd/conf/httpd.conf b/httpd/conf/httpd.conf
index 5f63fa1..a321c39 100644
--- a/httpd/conf/httpd.conf
+++ b/httpd/conf/httpd.conf
@@ -405,7 +405,6 @@ TraceEnable Off
         ProxyFCGIBackendType GENERIC
     </Directory>
     ProxyPass /.well-known !
-    ProxyPass /ghostblog balancer://proxy3000
     RewriteEngine on
     RewriteCond %{HTTP_HOST} =webmail.rocky9-pro.virtualmin.dev
     RewriteRule ^(?!/.well-known)(.*) https://rocky9-pro.virtualmin.dev:20000/ [R]
@@ -429,10 +428,6 @@ TraceEnable Off
         AuthUserFile /home/rocky9-pro/.awstats-htpasswd
         require valid-user
     </Files>
-    ProxyPassReverse /ghostblog balancer://proxy3000
-    <Proxy balancer://proxy3000>
-        BalancerMember http://127.0.0.1:3000/ghostblog
-    </Proxy>
1 Like

That seems like a bug! Were these missing from the port 80 or port 443 virtualhost?

Also, how was the upgrade done exactly?

Missing for both ports 80 and 443.

For the upgrade:

  • I went as usual to the System Settings in Virtualmin
  • Scrolled down to Web App Installers
  • Selected the Web App Summary tab
  • Saw that my version 6.13.2 was out-of-date and that 6.15.0 was offered
  • Clicked Ghost next to the first website in the list
  • Then clicked the Upgrade To Version button for 6.15.0
  • It showed the usual downloading and installing window
  • Then gave me the usual confirmation of a running updated Ghost website

This is my usual process for Ghost updates that I’ve been doing for a couple of years, and this is only the second hitch I’ve ever encountered when clicking the upgrade button (mostly I’m very happy and confident in clicking the button :slight_smile: )!

Is there a way for me manually to bring my website back online in the interim? As I detailed in my first post, I tried manually adding the missing proxy entries back in what appeared to be all the appropriate places, however still the Ghost website continues to generate Internal Server Errors.

Can you show the directives you tried to add that still didn’t work?

And did you restart Apache manually after editing its config?

I included all the directives I edited in my original post, and also mentioned there that I restarted Apache. Somewhere I missed something though, I wish I knew where! One thing that did seem problematic was that I couldn’t get the URL parameter for the proxy to ‘stick’, even though I saved it (and even though I tried modifying directly with modify-proxy, also detailed in my first post).

I carefully detailed the directives I changed in my OP (along with noting that I’d restarted Apache after doing so), but clearly I’d missed something, so I ended up deleting the website and reinstalling Ghost freshly. I may have lost a few things in the deletion/reinstallation process, but it’s better than continuing to have the website down for many days now.

Is there a reason that directives are deleted then replaced during a Ghost update? I’m a little afraid now of clicking the update button in Virtualmin. The alternative is doing manual updates on the command line, which I may do going forward, even though it’s more fiddly than using Virtualmin.

This is clearly a bug, and we need to fix it on our side. We’re sorry about that!

@Jamie, do you know what’s going on here?

That’s odd, because on an upgrade the proxy directives shouldn’t be changed.

Let me see if I can reproduce it..

Oh I see the bug here! This will be fixed in the next Virtualmin release.

1 Like

I saw that Virtualmin updated the other day (to 8.0.1 Pro on my side), so when I saw there were Ghost updates, I assumed it’d be safe to click the Update button. Sadly, my website went offline again, and I saw the installer deleting the proxy paths too :frowning:

I wish I knew how manually to fix this, rather than deleting the whole Ghost install and carefully restoring from backups (which is what I ended up doing last time, after not hearing back about which directives I’d missed when trying to bring my website back online as in my original post).

Damn, I thought we’d fixed this.

Where in the Virtualmin UI did you do the Ghost upgrade exactly?

Virtualmin 8.0.1 doesn’t include that patch Jamie implemented earlier and will only be available in the full Virtualmin 8.1.0 release.

For now, since Ghost moves very fast, you should apply the following Virtualmin patch to make sure it doesn’t remove the proxy path again:

webmin patch https://github.com/virtualmin/virtualmin-gpl/commit/6ec7f57