HTTP ERROR 503 Service Unavailable #9880
ISSUE TYPE
COMPONENT NAMECLOUDSTACK VERSIONCONFIGURATIONOS / ENVIRONMENTDebian 12.7 Stable SUMMARYSTEPS TO REPRODUCEEXPECTED RESULTSACTUAL RESULTS |
Answered by
penguin2716
Nov 6, 2024
Replies: 25 comments 29 replies
|
can you search |
0 replies
|
0 replies
0 replies
|
@gobiursus can you re-do all steps and share the output ? |
0 replies
|
How do I check the connection? How do I fix?
…On October 18, 2024 8:35:27 AM UTC, Wei Zhou ***@***.***> wrote:
@gobiursus
from my experience, it is mostly caused by db misconfiguration.
in short, the mgmt server cannot connect to the db
--
Reply to this email directly or view it on GitHub:
#9815 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
0 replies
|
There was no error during installation. |
0 replies
|
0 replies
|
I captured some extra stuff, I am not sure if they are related: |
0 replies
Logs look ok |
0 replies
|
I loaded the full log earlier in the link.
…On October 18, 2024 7:58:39 PM UTC, Wei Zhou ***@***.***> wrote:
> I captured some extra stuff, I am not sure if they are related:
> ```
> ● cloudstack-management.service - CloudStack Management Server
> Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; preset: enabled)
> Active: active (running) since Fri 2024-10-18 14:33:44 EDT; 1s ago
> Main PID: 20191 (java)
> Tasks: 30 (limit: 9471)
> Memory: 320.8M
> CPU: 1.919s
> CGroup: /system.slice/cloudstack-management.service
> └─20191 /usr/bin/java -Djava.security.properties=/etc/cloudstack/management/java.security.ciphers -Djava.awt.headless=true -Xmx2G -XX:+UseParallelGC -XX:MaxGCPauseMillis=500 -XX:+HeapDumpOnOutOfMem>
>
> Oct 18 14:33:44 manage systemd[1]: Started cloudstack-management.service - CloudStack Management Server.
> Oct 18 14:33:45 manage java[20191]: INFO [o.a.c.ServerDaemon] (main:null) (logid:) Server configuration file found: /etc/cloudstack/management/server.properties
> Oct 18 14:33:45 manage java[20191]: INFO [o.a.c.ServerDaemon] (main:null) (logid:) Initializing server daemon on null, with http.enable=true, http.port=8080, https.enable=false, https.port=8443, context.path=/>
> Oct 18 14:33:45 manage java[20191]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> Oct 18 14:33:45 manage java[20191]: SLF4J: Defaulting to no-operation (NOP) logger implementation
> Oct 18 14:33:45 manage java[20191]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details
> ```
Logs look ok
Can you load the full management-server.log ?
--
Reply to this email directly or view it on GitHub:
#9815 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
0 replies
|
<https://pastebin.com/R3CT2v5Y>
There you go
…On October 18, 2024 7:58:39 PM UTC, Wei Zhou ***@***.***> wrote:
> I captured some extra stuff, I am not sure if they are related:
> ```
> ● cloudstack-management.service - CloudStack Management Server
> Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; preset: enabled)
> Active: active (running) since Fri 2024-10-18 14:33:44 EDT; 1s ago
> Main PID: 20191 (java)
> Tasks: 30 (limit: 9471)
> Memory: 320.8M
> CPU: 1.919s
> CGroup: /system.slice/cloudstack-management.service
> └─20191 /usr/bin/java -Djava.security.properties=/etc/cloudstack/management/java.security.ciphers -Djava.awt.headless=true -Xmx2G -XX:+UseParallelGC -XX:MaxGCPauseMillis=500 -XX:+HeapDumpOnOutOfMem>
>
> Oct 18 14:33:44 manage systemd[1]: Started cloudstack-management.service - CloudStack Management Server.
> Oct 18 14:33:45 manage java[20191]: INFO [o.a.c.ServerDaemon] (main:null) (logid:) Server configuration file found: /etc/cloudstack/management/server.properties
> Oct 18 14:33:45 manage java[20191]: INFO [o.a.c.ServerDaemon] (main:null) (logid:) Initializing server daemon on null, with http.enable=true, http.port=8080, https.enable=false, https.port=8443, context.path=/>
> Oct 18 14:33:45 manage java[20191]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> Oct 18 14:33:45 manage java[20191]: SLF4J: Defaulting to no-operation (NOP) logger implementation
> Oct 18 14:33:45 manage java[20191]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details
> ```
Logs look ok
Can you load the full management-server.log ?
--
Reply to this email directly or view it on GitHub:
#9815 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
0 replies
|
I have the same issue after upgrading from 4.19.0.0 to 4.19.1.2 on Ubuntu 22.04. MySQL and the management server are running on the same node. After the following steps, the management console works now (although it may not be a good solution since the password is saved in plaintext). The problem may be related to somewhere around encryption / decryption logic?
--- db.properties.bak 2024-10-25 08:51:07.179157143 +0900
+++ db.properties 2024-10-25 08:50:10.391852007 +0900
@@ -22,7 +22,7 @@
# CloudStack database settings
db.cloud.username=cloud
-db.cloud.password=ENC(<encrypted_text>)
+db.cloud.password=plain_password_text
db.cloud.host=localhost
db.cloud.driver=jdbc:mysql
db.cloud.port=3306
@@ -53,7 +53,7 @@
# usage database settings
db.usage.username=cloud
-db.usage.password=ENC(<encrypted_text>)
+db.usage.password=plain_password_text
db.usage.host=localhost
db.usage.driver=jdbc:mysql
db.usage.port=3306
@@ -100,4 +100,4 @@
db.usage.secondsBeforeRetrySource=3600
db.usage.queriesBeforeRetrySource=5000
db.usage.initialTimeout=3600
-db.cloud.encryptor.version=V2
+#db.cloud.encryptor.version=V2 |
11 replies
|
When I run cloudstack-management 4.19.1.2 with java17, I got the following error |
So the problem was that I followed the |
|
right. the doc needs to be fixed. however, when you downgrade to java11, cloudstack should work well. the db.cloud.password and db.cloud.password are both decrypted as "password", are they correct ? |
Yes, the db.cloud.password and db.usage.password is both set to "password".
After setting default Java vertion to 11, the management server works well. |
Answer selected by
weizhouapache
It worked for me as well but SSO via SAML is not working after the upgrade. |
@Avneetdabas Can you upgrade to 4.19.1.3 ? |
I was having the exact same issue on AlmaLinux 9.3 and this resolved the problem as well. |
0 replies
@purteppers |
0 replies
Yes, specifically 8.0.36 |
0 replies
|
I suggest we shouldn't use MySQL from its official repo, we should use MariaDB from repo of Debian Stable.
…On October 30, 2024 4:02:10 PM UTC, purteppers ***@***.***> wrote:
> > I was having the exact same issue on AlmaLinux 9.3 and this resolved the problem as well.
>
> @purteppers do you use mysql 8.0 as well ?
Yes, specifically 8.0.36
--
Reply to this email directly or view it on GitHub:
#9815 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
1 reply
|
I tested upgrade on debian 12 and alma9 with mysql 8.0, both worked |
|
I did not find any valuable information in the log. is there anything wrong in the db upgrade ? cc @gobiursus @purteppers |
8 replies
Do you have the original db.properties ? |
|
Unfortunately since I was just testing the install I do not, I can run through the install again on a fresh Alma install if you want and provide the results. |
Ok I suspect it is caused by char # in the password |
|
The management service is currently working with the password with '#' and no encryption, so I'm not sure what changing the password will accomplish. Do you want me revert back to the encryption settings and change the password? |
|
ignore it, thanks anyway @purteppers |
|
How did you install your MySQL?Please don't use the MySQL from MySQL's official repo.
On Thursday, October 31, 2024 at 03:30:46 p.m. GMT, Wei Zhou ***@***.***> wrote:
I have tested debian/ubuntu/alma/rocky with different versions, cannot reproduce the issue unfortunately.
@purteppers
can you share the output of some commands ?
egrep 'secret|password' /etc/cloudstack/management/db.properties
cat /etc/cloudstack/management/key
# the logs when the issue is not fixed
egrep 'c.c.u.c.CloudStackEncryptor|c.c.u.Database' /var/log/cloudstack/management/management-server.log
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
1 reply
@gobiursus |
|
I installed MariaDB 10.11.6 from Debian 12 Stable repo byapt install mariadb-server
On Thursday, October 31, 2024 at 03:37:05 p.m. GMT, Wei Zhou ***@***.***> wrote:
How did you install your MySQL?Please don't use the MySQL from MySQL's official repo.
@gobiursus
I installed mysql 8.0.36 from the alma9/rocky9 official repo by
dnf install -y mysql-server
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
1 reply
I tried mariadb 10 on debian12 and ubuntu 24, also mysql 8.0.x on alma9/rocky9, no issues found Probably it is not related to mariadb/mysql, but some value in db.properties which cannot be decrypted. |
|
Java 17 is the default version of Debian 12.
Also the default database in Debian 12 is MariaDB not MySQL.
…On November 6, 2024 2:45:30 PM UTC, Wei Zhou ***@***.***> wrote:
@penguin2716
if you run with cloudstack 4.19 or prior, JAVA 17 is not supported.
JAVA 17 is supported since 4.20 (will be released in the coming weeks): #8497
--
Reply to this email directly or view it on GitHub:
#9880 (reply in thread)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
1 reply
yes @gobiursus |
|
@gobiursus @penguin2716 @purteppers |
1 reply
|
It looks reasonable. I just confirmed that the management server starts without 503 error on Debian 12.8 / Java 11.0.25+9 (Temurin) / MariaDB 10.11.6. |
|
|
0 replies
|
closed this for lack of interaction, please reopen or open a new item if applicable |
0 replies
|
I encounter this problem upgrading from 4.19.1.1 to 4.20.0. After upgrade I tried to access UI but got the 503 error. After downgrading by selecting java 11, I was able to access the UI again. I thought 4.20 needed java 17 to work?? Still searching with a solution. I revert my mgm server back to 4.19.1 java11 and all my KVM host are running on 4.20. All my system VM are also on 4.19.1. |
5 replies
4.20 works with both java 11 and java 17. java 17 is recommended. All mgmt server and kvm hosts should use the same ACS version. If java 17 does not work with ACS 4.20.0, can you share more logs ? |
|
/etc/default/cloudstack-management: Licensed to the Apache Software Foundation (ASF) under oneor more contributor license agreements. See the NOTICE filedistributed with this work for additional informationregarding copyright ownership. The ASF licenses this fileto you under the Apache License, Version 2.0 (the"License"); you may not use this file except in compliancewith the License. You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing,software distributed under the License is distributed on an"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANYKIND, either express or implied. See the License for thespecific language governing permissions and limitationsunder the License.JAVA_OPTS="-Djava.security.properties=/etc/cloudstack/management/java.security.ciphers -Djava.awt.headless=true -Dcom.sun.management.jmxremote=false -Xmx6G -XX:+UseParallelGC -XX:MaxGCPauseMillis=500 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:ErrorFile=/var/log/cloudstack/management/cloudstack-management.err " CLASSPATH="/usr/share/cloudstack-management/lib/:/etc/cloudstack/management:/usr/share/cloudstack-common:/usr/share/cloudstack-management/setup:/usr/share/cloudstack-management:/usr/share/java/mysql-connector-java.jar:/usr/share/cloudstack-mysql-ha/lib/" BOOTSTRAP_CLASS=org.apache.cloudstack.ServerDaemon ################################################################################################ This option here should be used with 'systemmd' based operating systems such as CentOS7, Ubuntu 16, and so on.#JAVA_DEBUG="-agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n" On the other hand, this option is used by CentOS6.#JAVA_DEBUG="-Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n" |
|
sorry, I have no idea why it came out with different front size. I just copy and paste. |
|
After upgrading to 4.20 with java17 root@xx-mgm-01:/var/log/cloudstack/management# tail -f management-server.log |
|
I also created an issue about this #10960 where I pasted more logs. |
|
I have seen this issue during CloudStack deployment. Check your cluster.node.ip (sudo cat /etc/cloudstack/management/db.properties) value. If its set to the loopback or a strange IP it can cause these issues. Update cluster.node.ip value to the IP of the CloudStack management server (ip a) and restart (sudo systemctl restart cloudstack-management). You should see clean logs after that (sudo grep -i error /var/log/cloudstack/management/management-server.log). |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Yes, the db.cloud.password and db.usage.password is both set to "password".
After setting default Java vertion to 11, the management server works well.