Fetch AI Outage History

**Firing** Value: B0=1 Labels: - alertname = Management PodCrashLooping - grafana_folder = Infra Monitoring Annotations: - summary = ethwatch-ethwatch-544cff579d-7cmzc Pod is in CLB in management Namespace Source: http://monitor.fetch-ai.com/alerting/grafana/ef9ffd4q26cqoa/view?orgId=1 Silence: http://monitor.fetch-ai.com/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Def9ffd4q26cqoa&orgId=1

**Firing** Value: [no value] Labels: - alertname = DatasourceNoData - datasource_uid = een50xvibpywwa - grafana_folder = Ledger Monitoring - ref_id = A - rulename = Alexandria Mainnet is Down Annotations: - description = Alexandria is down in Mainnet - summary = Alexandria is down in Mainnet Source: http://monitor.fetch-ai.com/alerting/grafana/ceo80hs8yg8aob/view?orgId=1 Silence: http://monitor.fetch-ai.com/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dceo80hs8yg8aob&matcher=datasource_uid%3Deen50xvibpywwa&matcher=ref_id%3DA&matcher=rulename%3DAlexandria+Mainnet+is+Down&orgId=1

**Firing** Value: B0=1, B1=1 Labels: - alertname = Mainnet PodCrashLooping - grafana_folder = Ledger Monitoring Annotations: - summary = alexandria-6985946b5d-dqz84 Pod is in CLB in alexandria Namespace cache-db-0 Pod is in CLB in alexandria Namespace Source: http://monitor.fetch-ai.com/alerting/grafana/df9ff666qb1tsa/view?orgId=1 Silence: http://monitor.fetch-ai.com/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Ddf9ff666qb1tsa&orgId=1

**Firing** Value: B0=1 Labels: - alertname = Prod-v2 PodCrashLooping - grafana_folder = Agentverse Monitoring Annotations: - summary = elastic-sync-29653935-khd7r Pod is in CLB in toddleragi Namespace Source: http://monitor.fetch-ai.com/alerting/grafana/bf9ff97rxfqbkb/view?orgId=1 Silence: http://monitor.fetch-ai.com/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dbf9ff97rxfqbkb&orgId=1

**Firing** Value: B0=0 Labels: - alertname = fetchhub-node-sync-rpc0 - grafana_folder = Ledger Monitoring Annotations: - summary = node-health-service.fetchhub-d.svc.cluster.local:80 is down Source: http://monitor.fetch-ai.com/alerting/grafana/deo7u6hbt5x4wf/view?orgId=1 Silence: http://monitor.fetch-ai.com/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Ddeo7u6hbt5x4wf&orgId=1

**Firing** Value: B0=99.91770833333334 Labels: - alertname = Agents Memory - grafana_folder = Agents Monitoring Annotations: - summary = agents / agents-postgres-db-1: 99.91770833333334% Source: http://monitor.fetch-ai.com/alerting/grafana/bfcn049lokwzka/view?orgId=1 Silence: http://monitor.fetch-ai.com/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dbfcn049lokwzka&orgId=1

**Firing** Value: B0=491 Labels: - alertname = Prod LLM Proxy broken pipe alert - grafana_folder = Infra Monitoring Annotations: - summary = write: broken pipe Error occurred in the last 5 min Source: http://monitor.fetch-ai.com/alerting/grafana/bfh686tibusqob/view?orgId=1 Silence: http://monitor.fetch-ai.com/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dbfh686tibusqob&orgId=1

**Firing** Value: B0=1 Labels: - alertname = Testnet Indexer CrashLoopBackOff pod alert - grafana_folder = Ledger Monitoring Annotations: - description = Pod [no value]/[no value] ([no value]) is in waiting state (reason: "CrashLoopBackOff"). - summary = Indexer Pod is crash looping. Source: http://monitor.fetch-ai.com/alerting/grafana/aeo87b93us1dse/view?orgId=1 Silence: http://monitor.fetch-ai.com/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Daeo87b93us1dse&orgId=1

**Firing** Value: B0=121 Labels: - alertname = Spike Unacknowledged Msg - grafana_folder = Infra Monitoring Annotations: - summary = agent_data_queue queue in toddleragi namespace Unacknowledged messages are above: 121% Source: http://monitor.fetch-ai.com/alerting/grafana/aeo46mdlhdhq8c/view?orgId=1 Silence: http://monitor.fetch-ai.com/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Daeo46mdlhdhq8c&orgId=1

We are currently investigating this issue.

We are currently investigating this issue.

Updated hosting-api. Interactions are temporarily unavailable until later update.

We are currently investigating this issue.

We are currently investigating this issue.

AI Engine / DeltaV / non agentverse agent shows agent offline/unreachable for Agentverse hosted agents. We are investigating, fix expected later on today.