安装完毕后无法访问或页面出现Internal Server Error

zw12078

请问部署完毕后不能访问，有什么解决办法
附发现的错误日志图：
1、ks-apigateway

2、ks-apiserver

3、ks-console

4、redis-ha-proxy

重新安装很多遍，都是这样，请大神指点；

zw12078

zw12078
补充POD状态

Cauchy

可以把redis那些pod重启下试试

zw12078

Cauchy 我试一下，

zw12078

Cauchy 重启了不管用，pod时间会不会是导致的问题，系统时间没问题，所有pod时间对不上

zheng1

你好，可以贴一下redis-ha-server这三个pod的日志吗？看上去和redis关系比较大

zw12078

zheng1
看着似乎没有问题

Jeff

所有节点时间是同步的么，容器里时间是UTC时间，与北京时间差8小时

zw12078

Jeff 时间是同步的

zw12078

登录页面是这个错误

hongming

@zw12078

kubectl -n kubesphere-system logs -l app=ks-console
kubectl -n kubesphere-system logs -l app=ks-apigateway

主要看看这两个组件有没有异常日志

zw12078

hongming

ks-console 日志：

   at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
(node:16) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 38)
(node:16) UnhandledPromiseRejectionWarning: redis.info: Executed timeout 5000 ms
    at Timeout._onTimeout (/root/KubeSphereUI/server/server.js:230:190110)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
(node:16) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 39)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
(node:16) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 26)
(node:16) UnhandledPromiseRejectionWarning: redis.info: Executed timeout 5000 ms
    at Timeout._onTimeout (/root/KubeSphereUI/server/server.js:230:190110)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
(node:16) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 27)
    at Timer.listOnTimeout (timers.js:290:5)
(node:16) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 45)
[ioredis] Unhandled error event: Error: read ECONNRESET
    at TCP.onread (net.js:622:25)
(node:16) UnhandledPromiseRejectionWarning: redis.info: Executed timeout 5000 ms
    at Timeout._onTimeout (/root/KubeSphereUI/server/server.js:230:190110)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
(node:16) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 46)

ks-apigateway日志：

2020/05/20 05:45:18 [INFO][cache:0xc000203400] Scanning for stale OCSP staples
2020/05/20 05:45:18 [INFO][cache:0xc000203400] Done checking OCSP staples
2020/05/20 05:51:29 [WARNING] Sending telemetry (attempt 1): Post https://telemetry.caddyserver.com/v1/update/dd8c378b-2ad9-4004-a7f8-ec2531374506: x509: certificate is valid for elves.live, not telemetry.caddyserver.com - backing off and retrying
2020/05/20 05:51:42 [WARNING] Sending telemetry (attempt 2): Post https://telemetry.caddyserver.com/v1/update/dd8c378b-2ad9-4004-a7f8-ec2531374506: dial tcp: lookup telemetry.caddyserver.com on 169.254.25.10:53: no such host - backing off and retrying
2020/05/20 05:52:09 [WARNING] Sending telemetry (attempt 3): Post https://telemetry.caddyserver.com/v1/update/dd8c378b-2ad9-4004-a7f8-ec2531374506: dial tcp [2604:a880:800:a1::953:e001]:443: connect: network is unreachable - backing off and retrying
2020/05/20 05:53:13 [ERROR] Sending telemetry: Post https://telemetry.caddyserver.com/v1/update/dd8c378b-2ad9-4004-a7f8-ec2531374506: dial tcp [2604:a880:800:a1::953:e001]:443: connect: network is unreachable
2020/05/20 04:50:15 [WARNING] Sending telemetry (attempt 1): Post https://telemetry.caddyserver.com/v1/update/0488c51f-3a97-49eb-adc1-44a73aa4d91a: x509: certificate is valid for elves.live, not telemetry.caddyserver.com - backing off and retrying
2020/05/20 04:50:28 [WARNING] Sending telemetry (attempt 2): Post https://telemetry.caddyserver.com/v1/update/0488c51f-3a97-49eb-adc1-44a73aa4d91a: dial tcp: lookup telemetry.caddyserver.com on 169.254.25.10:53: read udp 172.18.200.22:38861->169.254.25.10:53: i/o timeout - backing off and retrying
2020/05/20 04:50:56 [WARNING] Sending telemetry (attempt 3): Post https://telemetry.caddyserver.com/v1/update/0488c51f-3a97-49eb-adc1-44a73aa4d91a: x509: certificate is valid for elves.live, not telemetry.caddyserver.com - backing off and retrying
2020/05/20 04:52:00 [ERROR] Sending telemetry: Post https://telemetry.caddyserver.com/v1/update/0488c51f-3a97-49eb-adc1-44a73aa4d91a: x509: certificate is valid for elves.live, not telemetry.caddyserver.com
2020/05/20 05:45:18 [INFO][cache:0xc0001db2c0] Scanning for stale OCSP staples
2020/05/20 05:45:18 [INFO][cache:0xc0001db2c0] Done checking OCSP staples
2020/05/20 05:52:03 [WARNING] Sending telemetry (attempt 1): Post https://telemetry.caddyserver.com/v1/update/0488c51f-3a97-49eb-adc1-44a73aa4d91a: x509: certificate is valid for elves.live, not telemetry.caddyserver.com - backing off and retrying
2020/05/20 05:52:16 [WARNING] Sending telemetry (attempt 2): Post https://telemetry.caddyserver.com/v1/update/0488c51f-3a97-49eb-adc1-44a73aa4d91a: dial tcp: lookup telemetry.caddyserver.com on 169.254.25.10:53: read udp 172.18.200.22:33558->169.254.25.10:53: i/o timeout - backing off and retrying
2020/05/20 05:52:44 [WARNING] Sending telemetry (attempt 3): Post https://telemetry.caddyserver.com/v1/update/0488c51f-3a97-49eb-adc1-44a73aa4d91a: x509: certificate is valid for elves.live, not telemetry.caddyserver.com - backing off and retrying
2020/05/20 05:53:48 [ERROR] Sending telemetry: Post https://telemetry.caddyserver.com/v1/update/0488c51f-3a97-49eb-adc1-44a73aa4d91a: x509: certificate is valid for elves.live, not telemetry.caddyserver.com
2020/05/20 04:48:57 [WARNING] Sending telemetry (attempt 1): Post https://telemetry.caddyserver.com/v1/update/0a9f6eb6-3457-4b45-85a6-06ffa9f45c59: x509: certificate is valid for elves.live, not telemetry.caddyserver.com - backing off and retrying
2020/05/20 04:49:10 [WARNING] Sending telemetry (attempt 2): Post https://telemetry.caddyserver.com/v1/update/0a9f6eb6-3457-4b45-85a6-06ffa9f45c59: dial tcp: lookup telemetry.caddyserver.com on 169.254.25.10:53: read udp 172.18.200.24:53574->169.254.25.10:53: i/o timeout - backing off and retrying
2020/05/20 04:49:37 [WARNING] Sending telemetry (attempt 3): Post https://telemetry.caddyserver.com/v1/update/0a9f6eb6-3457-4b45-85a6-06ffa9f45c59: dial tcp: lookup telemetry.caddyserver.com on 169.254.25.10:53: no such host - backing off and retrying
2020/05/20 04:50:41 [ERROR] Sending telemetry: Post https://telemetry.caddyserver.com/v1/update/0a9f6eb6-3457-4b45-85a6-06ffa9f45c59: dial tcp: lookup telemetry.caddyserver.com on 169.254.25.10:53: no such host
2020/05/20 05:45:18 [INFO][cache:0xc0000cb090] Scanning for stale OCSP staples
2020/05/20 05:45:18 [INFO][cache:0xc0000cb090] Done checking OCSP staples
2020/05/20 05:51:07 [WARNING] Sending telemetry (attempt 1): Post https://telemetry.caddyserver.com/v1/update/0a9f6eb6-3457-4b45-85a6-06ffa9f45c59: dial tcp [2604:a880:800:a1::953:e001]:443: connect: network is unreachable - backing off and retrying
2020/05/20 05:51:16 [WARNING] Sending telemetry (attempt 2): Post https://telemetry.caddyserver.com/v1/update/0a9f6eb6-3457-4b45-85a6-06ffa9f45c59: x509: certificate is valid for elves.live, not telemetry.caddyserver.com - backing off and retrying
2020/05/20 05:51:43 [WARNING] Sending telemetry (attempt 3): Post https://telemetry.caddyserver.com/v1/update/0a9f6eb6-3457-4b45-85a6-06ffa9f45c59: x509: certificate is valid for elves.live, not telemetry.caddyserver.com - backing off and retrying
2020/05/20 05:52:47 [ERROR] Sending telemetry: Post https://telemetry.caddyserver.com/v1/update/0a9f6eb6-3457-4b45-85a6-06ffa9f45c59: dial tcp: lookup telemetry.caddyserver.com on 169.254.25.10:53: no such host

zheng1

zw12078 你的电脑可以远程吗？可以把登录方式发到 kubesphere@yunify.com

zw12078

zheng1 我又重新安装一边，稍后我把服务器地址和用户发给你

zw12078

zheng1 已发送邮箱

zheng1

zw12078
看上去是网络问题，原因还在排查

[root@master2 ~]# redis-cli -h 172.18.100.114
^C
[root@master2 ~]# redis-cli -h 172.18.100.114
^C
[root@master2 ~]# redis-cli -h 172.18.100.114
172.18.100.114:6379>

zw12078

这个ip，是pod地址，需要在pod内部

zheng1

zw12078 我现在把ks-console从replica=3调到了1，现在发现回包有问题。从master2到ks-console的请求来看，ks-console返回了200，但是master2没有收到返回的包

17:36:02.532978 IP (tos 0x0, ttl 64, id 58277, offset 0, flags [DF], proto TCP (6), length 60)
    172.18.200.1.53440 > 172.18.200.34.8000: Flags [S], cksum 0xe877 (incorrect -> 0xa3ce), seq 1281390003, win 28000, options [mss 1400,sackOK,TS val 110728726 ecr 0,nop,wscale 7], length 0
17:36:02.533598 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.18.200.34.8000 > 172.18.200.1.53440: Flags [S.], cksum 0x2796 (correct), seq 1469547501, ack 1281390004, win 28960, options [mss 1460,sackOK,TS val 110728717 ecr 110728726,nop,wscale 7], length 0
17:36:02.533652 IP (tos 0x0, ttl 64, id 58278, offset 0, flags [DF], proto TCP (6), length 52)
    172.18.200.1.53440 > 172.18.200.34.8000: Flags [.], cksum 0xe86f (incorrect -> 0xc6a6), seq 1, ack 1, win 219, options [nop,nop,TS val 110728727 ecr 110728717], length 0
17:36:02.533722 IP (tos 0x0, ttl 64, id 58279, offset 0, flags [DF], proto TCP (6), length 134)
    172.18.200.1.53440 > 172.18.200.34.8000: Flags [P.], cksum 0xe8c1 (incorrect -> 0xd395), seq 1:83, ack 1, win 219, options [nop,nop,TS val 110728727 ecr 110728717], length 82
17:36:02.534072 IP (tos 0x0, ttl 63, id 5600, offset 0, flags [DF], proto TCP (6), length 52)
    172.18.200.34.8000 > 172.18.200.1.53440: Flags [.], cksum 0xc64b (correct), seq 1, ack 83, win 227, options [nop,nop,TS val 110728718 ecr 110728727], length 0
17:36:02.538724 IP (tos 0x0, ttl 63, id 5601, offset 0, flags [DF], proto TCP (6), length 279)
    172.18.200.34.8000 > 172.18.200.1.53440: Flags [P.], cksum 0x91ce (correct), seq 1:228, ack 83, win 227, options [nop,nop,TS val 110728722 ecr 110728727], length 227
17:36:02.538766 IP (tos 0x0, ttl 64, id 58280, offset 0, flags [DF], proto TCP (6), length 52)
    172.18.200.1.53440 > 172.18.200.34.8000: Flags [.], cksum 0xe86f (incorrect -> 0xc55e), seq 83, ack 228, win 228, options [nop,nop,TS val 110728732 ecr 110728722], length 0
17:36:02.539103 IP (tos 0x0, ttl 64, id 58281, offset 0, flags [DF], proto TCP (6), length 139)
    172.18.200.1.53440 > 172.18.200.34.8000: Flags [P.], cksum 0xe8c6 (incorrect -> 0xd02e), seq 83:170, ack 228, win 228, options [nop,nop,TS val 110728733 ecr 110728722], length 87
17:36:02.740042 IP (tos 0x0, ttl 64, id 58282, offset 0, flags [DF], proto TCP (6), length 139)
    172.18.200.1.53440 > 172.18.200.34.8000: Flags [P.], cksum 0xe8c6 (incorrect -> 0xcf65), seq 83:170, ack 228, win 228, options [nop,nop,TS val 110728934 ecr 110728722], length 87
17:36:02.740400 IP (tos 0x0, ttl 63, id 5613, offset 0, flags [DF], proto TCP (6), length 64)
    172.18.200.34.8000 > 172.18.200.1.53440: Flags [.], cksum 0xcc94 (correct), seq 15496, ack 170, win 227, options [nop,nop,TS val 110728924 ecr 110728934,nop,nop,sack 1 {83:170}], length 0
17:36:07.543415 IP (tos 0x0, ttl 63, id 5619, offset 0, flags [DF], proto TCP (6), length 294)
    172.18.200.34.8000 > 172.18.200.1.53440: Flags [FP.], cksum 0xe3d4 (correct), seq 16884:17126, ack 170, win 227, options [nop,nop,TS val 110733727 ecr 110728934], length 242
17:36:07.543473 IP (tos 0x0, ttl 64, id 58283, offset 0, flags [DF], proto TCP (6), length 64)
    172.18.200.1.53440 > 172.18.200.34.8000: Flags [.], cksum 0xe87b (incorrect -> 0x3f76), seq 170, ack 228, win 236, options [nop,nop,TS val 110733737 ecr 110728722,nop,nop,sack 1 {16884:17127}], length 0

zw12078

zheng1 什么问题会导致这个情况呢

zheng1

zw12078 还在排查，可以再将master3和master1的密码也发到刚才的邮箱中吗？

zw12078

zheng1 ok，可以的

zw12078

zheng1 已发送