Woke up in the morning and my selfhosted Lemmy server was basically braindead. I installed it in a Proxmox lxct container using the ansible playbook. It spams my logs with this: ``` target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=8206186d-eaf9-486d-99ad-d9c5def188c8", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:08.001260Z WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::activities::community::announce::receive lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/community/announce.rs:46 lemmyohaaxyz-lemmy-1 | 1: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=dee66940-4048-4424-88c8-51cb58851eb0 lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=dee66940-4048-4424-88c8-51cb58851eb0", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:08.442900Z WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::activities::community::announce::receive lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/community/announce.rs:46 lemmyohaaxyz-lemmy-1 | 1: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=ba977ad6-03ea-46b3-a7d8-c0eb05355358 lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=ba977ad6-03ea-46b3-a7d8-c0eb05355358", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:10.626474Z WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum PageOrNote lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::objects::comment::verify lemmyohaaxyz-lemmy-1 | at crates/apub/src/objects/comment.rs:127 lemmyohaaxyz-lemmy-1 | 1: lemmy_apub::fetcher::post_or_comment::verify lemmyohaaxyz-lemmy-1 | at crates/apub/src/fetcher/post_or_comment.rs:68 lemmyohaaxyz-lemmy-1 | 2: lemmy_apub::activities::voting::vote::verify lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/voting/vote.rs:57 lemmyohaaxyz-lemmy-1 | 3: lemmy_apub::activities::community::announce::receive lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/community/announce.rs:144 lemmyohaaxyz-lemmy-1 | 4: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=0aeb2c01-23ac-492c-9693-3a9bb171a900 lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum PageOrNote, context: SpanTrace [{ target: "lemmy_apub::objects::comment", name: "verify", file: "crates/apub/src/objects/comment.rs", line: 127 }, { target: "lemmy_apub::fetcher::post_or_comment", name: "verify", file: "crates/apub/src/fetcher/post_or_comment.rs", line: 68 }, { target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=0aeb2c01-23ac-492c-9693-3a9bb171a900", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:14.796561Z WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum PageOrNote lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::activities::voting::vote::verify lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/voting/vote.rs:57 lemmyohaaxyz-lemmy-1 | 1: lemmy_apub::activities::community::announce::receive lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/community/announce.rs:144 lemmyohaaxyz-lemmy-1 | 2: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=552c4fb1-c65c-4220-8919-430243d720cd lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum PageOrNote, context: SpanTrace [{ target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=552c4fb1-c65c-4220-8919-430243d720cd", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:27:00.779726Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T05:30:09.026861Z WARN lemmy_server::root_span_builder: Timeout occurred while waiting for a slot to become available lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::objects::person::read_from_id lemmyohaaxyz-lemmy-1 | at crates/apub/src/objects/person.rs:66 lemmyohaaxyz-lemmy-1 | 1: lemmy_apub::fetcher::user_or_community::read_from_id lemmyohaaxyz-lemmy-1 | at crates/apub/src/fetcher/user_or_community.rs:47 lemmyohaaxyz-lemmy-1 | 2: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=0a4c7741-ce73-4108-a323-e7f5cc5b92ae lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_apub::objects::person", name: "read_from_id", file: "crates/apub/src/objects/person.rs", line: 66 }, { target: "lemmy_apub::fetcher::user_or_community", name: "read_from_id", file: "crates/apub/src/fetcher/user_or_community.rs", line: 47 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=0a4c7741-ce73-4108-a323-e7f5cc5b92ae", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T05:35:46.568991Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for hot ranks update: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T05:42:37.481187Z WARN activitypub_federation::activity_queue: Queueing activity https://lemmy.ohaa.xyz/activities/announce/df1ccbd5-3971-49e1-96b4-df6e73f39420 to https://pcglinks.com/inbox for retry after connection failure: Request error: error sending request for url (https://pcglinks.com/inbox): operation timed out. Sleeping for 216000s and trying again lemmyohaaxyz-lemmy-1 | 2023-08-22T05:42:51.370663Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T05:46:36.393798Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T05:53:44.661098Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for hot ranks update: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T06:00:21.684837Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T06:10:18.172432Z WARN lemmy_server::root_span_builder: Timeout occurred while waiting for a slot to become available lemmyohaaxyz-lemmy-1 | 0: lemmy_apub::objects::person::from_json lemmyohaaxyz-lemmy-1 | at crates/apub/src/objects/person.rs:134 lemmyohaaxyz-lemmy-1 | 1: lemmy_apub::activities::verify_person_in_community lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/mod.rs:62 lemmyohaaxyz-lemmy-1 | 2: lemmy_apub::activities::voting::vote::verify lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/voting/vote.rs:57 lemmyohaaxyz-lemmy-1 | 3: lemmy_apub::activities::community::announce::receive lemmyohaaxyz-lemmy-1 | at crates/apub/src/activities/community/announce.rs:144 lemmyohaaxyz-lemmy-1 | 4: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=b7159d57-eaae-4992-8d37-0ab87f9f3adb lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_apub::objects::person", name: "from_json", file: "crates/apub/src/objects/person.rs", line: 134 }, { target: "lemmy_apub::activities", name: "verify_person_in_community", file: "crates/apub/src/activities/mod.rs", line: 62 }, { target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=b7159d57-eaae-4992-8d37-0ab87f9f3adb", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | 2023-08-22T06:16:50.839271Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for active counts update: could not translate host name "postgres" to address: Try again lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | 2023-08-22T06:26:51.471879Z WARN lemmy_server::root_span_builder: Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | Caused by: lemmyohaaxyz-lemmy-1 | Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT lemmyohaaxyz-lemmy-1 | 0: lemmy_server::root_span_builder::HTTP request lemmyohaaxyz-lemmy-1 | with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=57b8b802-fc7e-4f14-87ac-89de2b0d7770 lemmyohaaxyz-lemmy-1 | at src/root_span_builder.rs:16 lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT lemmyohaaxyz-lemmy-1 | lemmyohaaxyz-lemmy-1 | Caused by: lemmyohaaxyz-lemmy-1 | Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=57b8b802-fc7e-4f14-87ac-89de2b0d7770", file: "src/root_span_builder.rs", line: 16 }] } lemmyohaaxyz-lemmy-1 | thread 'actix-server worker 5' panicked at 'read local site data: LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=ab4b5ccf-f8b9-4acd-9e4b-3c7966a17fd5", file: "src/root_span_builder.rs", line: 16 }] }', crates/apub/src/lib.rs:45:8 lemmyohaaxyz-lemmy-1 | note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace lemmyohaaxyz-lemmy-1 | thread 'actix-server worker 3' panicked at 'read local site data: LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=4e4ba6d1-d716-4d48-93de-e3d7772243d7", file: "src/root_span_builder.rs", line: 16 }] }', crates/apub/src/lib.rs:45:8 lemmyohaaxyz-lemmy-1 | connection error: db error: FATAL: terminating connection due to administrator command lemmyohaaxyz-lemmy-1 | federation enabled, host is lemmy.ohaa.xyz lemmyohaaxyz-lemmy-1 | Starting http server at 0.0.0.0:8536 lemmyohaaxyz-lemmy-1 | connection error: db error: FATAL: terminating connection due to administrator command lemmyohaaxyz-lemmy-1 | federation enabled, host is lemmy.ohaa.xyz lemmyohaaxyz-lemmy-1 | Starting http server at 0.0.0.0:8536 ```