refactor: optimize event processor

Co-authored-by: Haobo Zhang <43698160+haobo8@users.noreply.github.com>

.gitignore

@@ -66,4 +66,6 @@ queries.active
 /n9e-*
 n9e.sql
 
-!/datasource
+!/datasource
+
+.env.json

README.md

@@ -3,7 +3,7 @@
     <img src="doc/img/Nightingale_L_V.png" alt="nightingale - cloud native monitoring" width="100" /></a>
 </p>
 <p align="center">
-  <b>开源告警管理专家 一体化的可观测平台</b>
+  <b>开源告警管理专家</b>
 </p>
 
 <p align="center">
@@ -27,77 +27,86 @@
 
 [English](./README_en.md) | [中文](./README.md)
 
-## 夜莺 Nightingale 是什么
+## 夜莺是什么
 
-> 夜莺 Nightingale 是什么，解决什么问题？以大家都很熟悉的 Grafana 做个类比，Grafana 擅长对接各种各样的数据源，然后提供灵活、强大、好看的可视化面板。夜莺则擅长对接各种多样的数据源，提供灵活、强大、高效的监控告警管理能力。从发展路径和定位来说，夜莺和 Grafana 很像，可以总结为一句话：可视化就用 Grafana，监控告警就找夜莺。
->
-> 在可视化领域，Grafana 是毫无争议的领导者，Grafana 在影响力、装机量、用户群、开发者数量等各个维度的数字上，相比夜莺都是追赶的榜样。巨无霸往往都是从一个切入点打开局面的，Grafana Labs 有了在可视化领域 Grafana 这个王牌，逐步扩展到整个可观测性方向，比如 Logging 维度有 Loki，Tracing 维度有 Tempo，Profiling 维度有收购来的 Pyroscope，On-call 维度有同样是收购来的 Grafana-OnCall 项目，还有时序数据库 Mimir、eBPF 采集器 Beyla、OpenTelemetry 采集器 Alloy、前端监控 SDK Faro，最终构成了一个完整的可观测性工具矩阵，但整个飞轮都是从 Grafana 项目开始转动起来的。
->
->夜莺，则是从监控告警这个切入点打开局面，也逐步横向做了相应扩展，比如夜莺也自研了可视化面板，如果你想有一个 all-in-one 的监控告警+可视化的工具，那么用夜莺也是正确的选择；比如 OnCall 方向，夜莺可以和 [Flashduty SaaS](https://flashcat.cloud/product/flashcat-duty/) 服务无缝的集成；在采集器方向，夜莺有配套的 [Categraf](https://flashcat.cloud/product/categraf)，可以一个采集器中管理所有的 exporter，并同时支持指标和日志的采集，极大减轻工程师维护的采集器数量和工作量（这个点太痛了，你可能也遇到过业务团队吐槽采集器数量比业务应用进程数量还多的窘况吧）。
+夜莺监控（Nightingale）是一款侧重告警的监控类开源项目。类似 Grafana 的数据源集成方式，夜莺也是对接多种既有的数据源，不过 Grafana 侧重在可视化，夜莺是侧重在告警引擎、告警事件的处理和分发。
 
-夜莺 Nightingale 作为一款开源云原生监控工具，最初由滴滴开发和开源，并于 2022 年 5 月 11 日，捐赠予中国计算机学会开源发展委员会（CCF ODC），为 CCF ODC 成立后接受捐赠的第一个开源项目。在 GitHub 上有超过 10000 颗星，是广受关注和使用的开源监控工具。夜莺的核心研发团队，也是 Open-Falcon 项目原核心研发人员，从 2014 年（Open-Falcon 是 2014 年开源）算起来，也有 10 年了，只为把监控做到极致。
+夜莺监控项目，最初由滴滴开发和开源，并于 2022 年 5 月 11 日，捐赠予中国计算机学会开源发展委员会（CCF ODC），为 CCF ODC 成立后接受捐赠的第一个开源项目。
 
+## 夜莺的工作逻辑
 
-## 快速开始
-- 👉 [文档中心](https://flashcat.cloud/docs/) | [下载中心](https://flashcat.cloud/download/nightingale/)
-- ❤️ [报告 Bug](https://github.com/ccfos/nightingale/issues/new?assignees=&labels=&projects=&template=question.yml)
-- ℹ️ 为了提供更快速的访问体验，上述文档和下载站点托管于 [FlashcatCloud](https://flashcat.cloud)
-- 💡 前后端代码分离，前端代码仓库：[https://github.com/n9e/fe](https://github.com/n9e/fe)
+很多用户已经自行采集了指标、日志数据，此时就把存储库（VictoriaMetrics、ElasticSearch等）作为数据源接入夜莺，即可在夜莺里配置告警规则、通知规则，完成告警事件的生成和派发。
 
-## 功能特点
+![夜莺产品架构](doc/img/readme/20240221152601.png)
 
-- 对接多种时序库，实现统一监控告警管理：支持对接的时序库包括 Prometheus、VictoriaMetrics、Thanos、Mimir、M3DB、TDengine 等。
-- 对接日志库，实现针对日志的监控告警：支持对接的日志库包括 ElasticSearch、Loki 等。
-- 专业告警能力：内置支持多种告警规则，可以扩展支持常见通知媒介，支持告警屏蔽/抑制/订阅/自愈、告警事件管理。
-- 高性能可视化引擎：支持多种图表样式，内置众多 Dashboard 模版，也可导入 Grafana 模版，开箱即用，开源协议商业友好。
-- 支持常见采集器：支持 [Categraf](https://flashcat.cloud/product/categraf)、Telegraf、Grafana-agent、Datadog-agent、各种 Exporter 作为采集器，没有什么数据是不能监控的。
-- 👀无缝搭配 [Flashduty](https://flashcat.cloud/product/flashcat-duty/)：实现告警聚合收敛、认领、升级、排班、IM集成，确保告警处理不遗漏，减少打扰，高效协同。
+夜莺项目本身不提供监控数据采集能力。推荐您使用 [Categraf](https://github.com/flashcatcloud/categraf) 作为采集器，可以和夜莺丝滑对接。
 
+[Categraf](https://github.com/flashcatcloud/categraf) 可以采集操作系统、网络设备、各类中间件、数据库的监控数据，通过 Remote Write 协议推送给夜莺，夜莺把监控数据转存到时序库（如 Prometheus、VictoriaMetrics 等），并提供告警和可视化能力。
 
-## 截图演示
-
-
-你可以在页面的右上角，切换语言和主题，目前我们支持英语、简体中文、繁体中文。
-
-![语言切换](doc/img/readme/n9e-switch-i18n.png)
-
-即时查询，类似 Prometheus 内置的查询分析页面，做 ad-hoc 查询，夜莺做了一些 UI 优化，同时提供了一些内置 promql 指标，让不太了解 promql 的用户也可以快速查询。
-
-![即时查询](doc/img/readme/20240513103305.png)
-
-当然，也可以直接通过指标视图查看，有了指标视图，即时查询基本可以不用了，或者只有高端玩家使用即时查询，普通用户直接通过指标视图查询即可。
-
-![指标视图](doc/img/readme/20240513103530.png)
-
-夜莺内置了常用仪表盘，可以直接导入使用。也可以导入 Grafana 仪表盘，不过只能兼容 Grafana 基本图表，如果已经习惯了 Grafana 建议继续使用 Grafana 看图，把夜莺作为一个告警引擎使用。
-
-![内置仪表盘](doc/img/readme/20240513103628.png)
-
-除了内置的仪表盘，也内置了很多告警规则，开箱即用。
-
-![内置告警规则](doc/img/readme/20240513103825.png)
-
-
-
-## 产品架构
-
-社区使用夜莺最多的场景就是使用夜莺做告警引擎，对接多套时序库，统一告警规则管理。绘图仍然使用 Grafana 居多。作为一个告警引擎，夜莺的产品架构如下：
-
-![产品架构](doc/img/readme/20240221152601.png)
-
-对于个别边缘机房，如果和中心夜莺服务端网络链路不好，希望提升告警可用性，我们也提供边缘机房告警引擎下沉部署模式，这个模式下，即便网络割裂，告警功能也不受影响。
+对于个别边缘机房，如果和中心夜莺服务端网络链路不好，希望提升告警可用性，夜莺也提供边缘机房告警引擎下沉部署模式，这个模式下，即便边缘和中心端网络割裂，告警功能也不受影响。
 
 ![边缘部署模式](doc/img/readme/20240222102119.png)
 
+> 上图中，机房A和中心机房的网络链路很好，所以直接由中心端的夜莺进程做告警引擎，机房B和中心机房的网络链路不好，所以在机房B部署了 `n9e-edge` 做告警引擎，对机房B的数据源做告警判定。
 
-## 交流渠道
-- 报告Bug，优先推荐提交[夜莺GitHub Issue](https://github.com/ccfos/nightingale/issues/new?assignees=&labels=kind%2Fbug&projects=&template=bug_report.yml)
-- 推荐完整浏览[夜莺文档站点](https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/introduction/)，了解更多信息
-- 加我微信：`picobyte`（我已关闭好友验证）拉入微信群，备注：`夜莺互助群`
+## 告警降噪、升级、协同
+
+夜莺的侧重点是做告警引擎，即负责产生告警事件，并根据规则做灵活派发，内置支持 20 种通知媒介（电话、短信、邮件、钉钉、飞书、企微、Slack 等）。
+
+如果您有更高级的需求，比如：
+
+- 想要把公司的多套监控系统产生的事件聚拢到一个平台，统一做收敛降噪、响应处理、数据分析
+- 想要支持人员的排班，践行 On-call 文化，想要支持告警认领、升级（避免遗漏）、协同处理
+
+那夜莺是不合适的，推荐您选用 [FlashDuty](https://flashcat.cloud/product/flashcat-duty/) 这样的 On-call 产品，产品简单易用，也有免费套餐。
+
+
+## 相关资料 & 交流渠道
+- 📚 [夜莺介绍PPT](https://mp.weixin.qq.com/s/Mkwx_46xrltSq8NLqAIYow) 对您了解夜莺各项关键特性会有帮助（PPT链接在文末）
+- 👉 [文档中心](https://flashcat.cloud/docs/) 为了更快的访问速度，站点托管在 [FlashcatCloud](https://flashcat.cloud)
+- ❤️ [报告 Bug](https://github.com/ccfos/nightingale/issues/new?assignees=&labels=&projects=&template=question.yml) 写清楚问题描述、复现步骤、截图等信息，更容易得到答案
+- 💡 前后端代码分离，前端代码仓库：[https://github.com/n9e/fe](https://github.com/n9e/fe)
+- 🎯 关注[这个公众号](https://gitlink.org.cn/UlricQin)了解更多夜莺动态和知识
+- 🌟 加我微信：`picobyte`（我已关闭好友验证）拉入微信群，备注：`夜莺互助群`，如果已经把夜莺上到生产环境，可联系我拉入资深监控用户群
+
+
+## 关键特性简介
+
+![夜莺告警规则](doc/img/readme/2025-05-23_18-43-37.png)
+
+- 夜莺支持告警规则、屏蔽规则、订阅规则、通知规则，内置支持 20 种通知媒介，支持消息模板自定义
+- 支持事件管道，对告警事件做 Pipeline 处理，方便和自有系统做自动化整合，比如给告警事件附加一些元信息，对事件做 relabel
+- 支持业务组概念，引入权限体系，分门别类管理各类规则
+- 很多数据库、中间件内置了告警规则，可以直接导入使用，也可以直接导入 Prometheus 的告警规则
+- 支持告警自愈，即告警之后自动触发一个脚本执行一些预定义的逻辑，比如清理一下磁盘、抓一下现场等
+
+![夜莺事件大盘](doc/img/readme/2025-05-30_08-49-28.png)
+
+- 夜莺存档了历史告警事件，支持多维度的查询和统计
+- 支持灵活的聚合分组，一目了然看到公司的告警事件分布情况
+
+![夜莺集成中心](doc/img/readme/2025-05-23_18-46-06.png)
+
+- 夜莺内置常用操作系统、中间件、数据库的的指标说明、仪表盘、告警规则，不过都是社区贡献的，整体也是参差不齐
+- 夜莺直接接收 Remote Write、OpenTSDB、Datadog、Falcon 等多种协议的数据，故而可以和各类 Agent 对接
+- 夜莺支持 Prometheus、ElasticSearch、Loki、TDEngine 等多种数据源，可以对其中的数据做告警
+- 夜莺可以很方便内嵌企业内部系统，比如 Grafana、CMDB 等，甚至可以配置这些内嵌系统的菜单可见性
+
+
+![夜莺仪表盘](doc/img/readme/2025-05-23_18-49-02.png)
+
+- 夜莺支持仪表盘功能，支持常见的图表类型，也内置了一些仪表盘，上图是其中一个仪表盘的截图。
+- 如果你已经习惯了 Grafana，建议仍然使用 Grafana 看图。Grafana 在看图方面道行更深。
+- 机器相关的监控数据，如果是 Categraf 采集的，建议使用夜莺自带的仪表盘查看，因为 Categraf 的指标命名 Follow 的是 Telegraf 的命名方式，和 Node Exporter 不同
+- 因为夜莺有个业务组的概念，机器可以归属不同的业务组，有时在仪表盘里只想查看当前所属业务组的机器，所以夜莺的仪表盘可以和业务组联动
 
 ## 广受关注
 [![Stargazers over time](https://api.star-history.com/svg?repos=ccfos/nightingale&type=Date)](https://star-history.com/#ccfos/nightingale&Date)
 
+## 感谢众多企业的信赖
+
+![夜莺客户](doc/img/readme/logos.png)
+
 ## 社区共建
 - ❇️ 请阅读浏览[夜莺开源项目和社区治理架构草案](./doc/community-governance.md)，真诚欢迎每一位用户、开发者、公司以及组织，使用夜莺监控、积极反馈 Bug、提交功能需求、分享最佳实践，共建专业、活跃的夜莺开源社区。
 - ❤️ 夜莺贡献者

alert/alert.go

@@ -64,6 +64,9 @@ func Initialize(configDir string, cryptoKey string) (func(), error) {
 	userGroupCache := memsto.NewUserGroupCache(ctx, syncStats)
 	taskTplsCache := memsto.NewTaskTplCache(ctx)
 	configCvalCache := memsto.NewCvalCache(ctx, syncStats)
+	notifyRuleCache := memsto.NewNotifyRuleCache(ctx, syncStats)
+	notifyChannelCache := memsto.NewNotifyChannelCache(ctx, syncStats)
+	messageTemplateCache := memsto.NewMessageTemplateCache(ctx, syncStats)
 
 	promClients := prom.NewPromClient(ctx)
 	dispatch.InitRegisterQueryFunc(promClients)
@@ -72,7 +75,7 @@ func Initialize(configDir string, cryptoKey string) (func(), error) {
 
 	macros.RegisterMacro(macros.MacroInVain)
 	dscache.Init(ctx, false)
-	Start(config.Alert, config.Pushgw, syncStats, alertStats, externalProcessors, targetCache, busiGroupCache, alertMuteCache, alertRuleCache, notifyConfigCache, taskTplsCache, dsCache, ctx, promClients, userCache, userGroupCache)
+	Start(config.Alert, config.Pushgw, syncStats, alertStats, externalProcessors, targetCache, busiGroupCache, alertMuteCache, alertRuleCache, notifyConfigCache, taskTplsCache, dsCache, ctx, promClients, userCache, userGroupCache, notifyRuleCache, notifyChannelCache, messageTemplateCache)
 
 	r := httpx.GinEngine(config.Global.RunMode, config.HTTP,
 		configCvalCache.PrintBodyPaths, configCvalCache.PrintAccessLog)
@@ -95,12 +98,14 @@ func Initialize(configDir string, cryptoKey string) (func(), error) {
 
 func Start(alertc aconf.Alert, pushgwc pconf.Pushgw, syncStats *memsto.Stats, alertStats *astats.Stats, externalProcessors *process.ExternalProcessorsType, targetCache *memsto.TargetCacheType, busiGroupCache *memsto.BusiGroupCacheType,
 	alertMuteCache *memsto.AlertMuteCacheType, alertRuleCache *memsto.AlertRuleCacheType, notifyConfigCache *memsto.NotifyConfigCacheType, taskTplsCache *memsto.TaskTplCache, datasourceCache *memsto.DatasourceCacheType, ctx *ctx.Context,
-	promClients *prom.PromClientMap, userCache *memsto.UserCacheType, userGroupCache *memsto.UserGroupCacheType) {
+	promClients *prom.PromClientMap, userCache *memsto.UserCacheType, userGroupCache *memsto.UserGroupCacheType, notifyRuleCache *memsto.NotifyRuleCacheType, notifyChannelCache *memsto.NotifyChannelCacheType, messageTemplateCache *memsto.MessageTemplateCacheType) {
 	alertSubscribeCache := memsto.NewAlertSubscribeCache(ctx, syncStats)
 	recordingRuleCache := memsto.NewRecordingRuleCache(ctx, syncStats)
 	targetsOfAlertRulesCache := memsto.NewTargetOfAlertRuleCache(ctx, alertc.Heartbeat.EngineName, syncStats)
 
 	go models.InitNotifyConfig(ctx, alertc.Alerting.TemplatesDir)
+	go models.InitNotifyChannel(ctx)
+	go models.InitMessageTemplate(ctx)
 
 	naming := naming.NewNaming(ctx, alertc.Heartbeat, alertStats)
 
@@ -110,7 +115,9 @@ func Start(alertc aconf.Alert, pushgwc pconf.Pushgw, syncStats *memsto.Stats, al
 	eval.NewScheduler(alertc, externalProcessors, alertRuleCache, targetCache, targetsOfAlertRulesCache,
 		busiGroupCache, alertMuteCache, datasourceCache, promClients, naming, ctx, alertStats)
 
-	dp := dispatch.NewDispatch(alertRuleCache, userCache, userGroupCache, alertSubscribeCache, targetCache, notifyConfigCache, taskTplsCache, alertc.Alerting, ctx, alertStats)
+	eventProcessorCache := memsto.NewEventProcessorCache(ctx, syncStats)
+
+	dp := dispatch.NewDispatch(alertRuleCache, userCache, userGroupCache, alertSubscribeCache, targetCache, notifyConfigCache, taskTplsCache, notifyRuleCache, notifyChannelCache, messageTemplateCache, eventProcessorCache, alertc.Alerting, ctx, alertStats)
 	consumer := dispatch.NewConsumer(alertc.Alerting, ctx, dp, promClients)
 
 	notifyRecordComsumer := sender.NewNotifyRecordConsumer(ctx)

alert/astats/stats.go

@@ -17,12 +17,15 @@ type Stats struct {
 	CounterRuleEval             *prometheus.CounterVec
 	CounterQueryDataErrorTotal  *prometheus.CounterVec
 	CounterQueryDataTotal       *prometheus.CounterVec
+	CounterVarFillingQuery      *prometheus.CounterVec
 	CounterRecordEval           *prometheus.CounterVec
 	CounterRecordEvalErrorTotal *prometheus.CounterVec
 	CounterMuteTotal            *prometheus.CounterVec
 	CounterRuleEvalErrorTotal   *prometheus.CounterVec
 	CounterHeartbeatErrorTotal  *prometheus.CounterVec
 	CounterSubEventTotal        *prometheus.CounterVec
+	GaugeQuerySeriesCount       *prometheus.GaugeVec
+	GaugeRuleEvalDuration       *prometheus.GaugeVec
 	GaugeNotifyRecordQueueSize  prometheus.Gauge
 }
 
@@ -53,7 +56,7 @@ func NewSyncStats() *Stats {
 		Subsystem: subsystem,
 		Name:      "query_data_total",
 		Help:      "Number of rule eval query data.",
-	}, []string{"datasource"})
+	}, []string{"datasource", "rule_id"})
 
 	CounterRecordEval := prometheus.NewCounterVec(prometheus.CounterOpts{
 		Namespace: namespace,
@@ -104,7 +107,7 @@ func NewSyncStats() *Stats {
 		Subsystem: subsystem,
 		Name:      "mute_total",
 		Help:      "Number of mute.",
-	}, []string{"group"})
+	}, []string{"group", "rule_id", "mute_rule_id", "datasource_id"})
 
 	CounterSubEventTotal := prometheus.NewCounterVec(prometheus.CounterOpts{
 		Namespace: namespace,
@@ -120,6 +123,12 @@ func NewSyncStats() *Stats {
 		Help:      "Number of heartbeat error.",
 	}, []string{})
 
+	GaugeQuerySeriesCount := prometheus.NewGaugeVec(prometheus.GaugeOpts{
+		Namespace: namespace,
+		Subsystem: subsystem,
+		Name:      "eval_query_series_count",
+		Help:      "Number of series retrieved from data source after query.",
+	}, []string{"rule_id", "datasource_id", "ref"})
 	// 通知记录队列的长度
 	GaugeNotifyRecordQueueSize := prometheus.NewGauge(prometheus.GaugeOpts{
 		Namespace: namespace,
@@ -128,6 +137,20 @@ func NewSyncStats() *Stats {
 		Help:      "The size of notify record queue.",
 	})
 
+	GaugeRuleEvalDuration := prometheus.NewGaugeVec(prometheus.GaugeOpts{
+		Namespace: namespace,
+		Subsystem: subsystem,
+		Name:      "rule_eval_duration_ms",
+		Help:      "Duration of rule eval in milliseconds.",
+	}, []string{"rule_id", "datasource_id"})
+
+	CounterVarFillingQuery := prometheus.NewCounterVec(prometheus.CounterOpts{
+		Namespace: namespace,
+		Subsystem: subsystem,
+		Name:      "var_filling_query_total",
+		Help:      "Number of var filling query.",
+	}, []string{"rule_id", "datasource_id", "ref", "typ"})
+
 	prometheus.MustRegister(
 		CounterAlertsTotal,
 		GaugeAlertQueueSize,
@@ -142,7 +165,10 @@ func NewSyncStats() *Stats {
 		CounterRuleEvalErrorTotal,
 		CounterHeartbeatErrorTotal,
 		CounterSubEventTotal,
+		GaugeQuerySeriesCount,
+		GaugeRuleEvalDuration,
 		GaugeNotifyRecordQueueSize,
+		CounterVarFillingQuery,
 	)
 
 	return &Stats{
@@ -159,6 +185,9 @@ func NewSyncStats() *Stats {
 		CounterRuleEvalErrorTotal:   CounterRuleEvalErrorTotal,
 		CounterHeartbeatErrorTotal:  CounterHeartbeatErrorTotal,
 		CounterSubEventTotal:        CounterSubEventTotal,
+		GaugeQuerySeriesCount:       GaugeQuerySeriesCount,
+		GaugeRuleEvalDuration:       GaugeRuleEvalDuration,
 		GaugeNotifyRecordQueueSize:  GaugeNotifyRecordQueueSize,
+		CounterVarFillingQuery:      CounterVarFillingQuery,
 	}
 }

alert/dispatch/dispatch.go

@@ -3,6 +3,8 @@ package dispatch
 import (
 	"bytes"
 	"encoding/json"
+	"errors"
+	"fmt"
 	"html/template"
 	"net/url"
 	"strconv"
@@ -13,6 +15,7 @@ import (
 	"github.com/ccfos/nightingale/v6/alert/aconf"
 	"github.com/ccfos/nightingale/v6/alert/astats"
 	"github.com/ccfos/nightingale/v6/alert/common"
+	"github.com/ccfos/nightingale/v6/alert/pipeline"
 	"github.com/ccfos/nightingale/v6/alert/sender"
 	"github.com/ccfos/nightingale/v6/memsto"
 	"github.com/ccfos/nightingale/v6/models"
@@ -30,6 +33,11 @@ type Dispatch struct {
 	notifyConfigCache   *memsto.NotifyConfigCacheType
 	taskTplsCache       *memsto.TaskTplCache
 
+	notifyRuleCache      *memsto.NotifyRuleCacheType
+	notifyChannelCache   *memsto.NotifyChannelCacheType
+	messageTemplateCache *memsto.MessageTemplateCacheType
+	eventProcessorCache  *memsto.EventProcessorCacheType
+
 	alerting aconf.Alerting
 
 	Senders          map[string]sender.Sender
@@ -47,15 +55,20 @@ type Dispatch struct {
 // 创建一个 Notify 实例
 func NewDispatch(alertRuleCache *memsto.AlertRuleCacheType, userCache *memsto.UserCacheType, userGroupCache *memsto.UserGroupCacheType,
 	alertSubscribeCache *memsto.AlertSubscribeCacheType, targetCache *memsto.TargetCacheType, notifyConfigCache *memsto.NotifyConfigCacheType,
-	taskTplsCache *memsto.TaskTplCache, alerting aconf.Alerting, ctx *ctx.Context, astats *astats.Stats) *Dispatch {
+	taskTplsCache *memsto.TaskTplCache, notifyRuleCache *memsto.NotifyRuleCacheType, notifyChannelCache *memsto.NotifyChannelCacheType,
+	messageTemplateCache *memsto.MessageTemplateCacheType, eventProcessorCache *memsto.EventProcessorCacheType, alerting aconf.Alerting, ctx *ctx.Context, astats *astats.Stats) *Dispatch {
 	notify := &Dispatch{
-		alertRuleCache:      alertRuleCache,
-		userCache:           userCache,
-		userGroupCache:      userGroupCache,
-		alertSubscribeCache: alertSubscribeCache,
-		targetCache:         targetCache,
-		notifyConfigCache:   notifyConfigCache,
-		taskTplsCache:       taskTplsCache,
+		alertRuleCache:       alertRuleCache,
+		userCache:            userCache,
+		userGroupCache:       userGroupCache,
+		alertSubscribeCache:  alertSubscribeCache,
+		targetCache:          targetCache,
+		notifyConfigCache:    notifyConfigCache,
+		taskTplsCache:        taskTplsCache,
+		notifyRuleCache:      notifyRuleCache,
+		notifyChannelCache:   notifyChannelCache,
+		messageTemplateCache: messageTemplateCache,
+		eventProcessorCache:  eventProcessorCache,
 
 		alerting: alerting,
 
@@ -67,11 +80,13 @@ func NewDispatch(alertRuleCache *memsto.AlertRuleCacheType, userCache *memsto.Us
 		ctx:    ctx,
 		Astats: astats,
 	}
+
+	pipeline.Init()
 	return notify
 }
 
 func (e *Dispatch) ReloadTpls() error {
-	err := e.relaodTpls()
+	err := e.reloadTpls()
 	if err != nil {
 		logger.Errorf("failed to reload tpls: %v", err)
 	}
@@ -79,13 +94,13 @@ func (e *Dispatch) ReloadTpls() error {
 	duration := time.Duration(9000) * time.Millisecond
 	for {
 		time.Sleep(duration)
-		if err := e.relaodTpls(); err != nil {
+		if err := e.reloadTpls(); err != nil {
 			logger.Warning("failed to reload tpls:", err)
 		}
 	}
 }
 
-func (e *Dispatch) relaodTpls() error {
+func (e *Dispatch) reloadTpls() error {
 	tmpTpls, err := models.ListTpls(e.ctx)
 	if err != nil {
 		return err
@@ -131,6 +146,354 @@ func (e *Dispatch) relaodTpls() error {
 	return nil
 }
 
+func (e *Dispatch) HandleEventWithNotifyRule(eventOrigin *models.AlertCurEvent) {
+
+	if len(eventOrigin.NotifyRuleIds) > 0 {
+		for _, notifyRuleId := range eventOrigin.NotifyRuleIds {
+			// 深拷贝新的 event，避免并发修改 event 冲突
+			eventCopy := eventOrigin.DeepCopy()
+
+			logger.Infof("notify rule ids: %v, event: %+v", notifyRuleId, eventCopy)
+			notifyRule := e.notifyRuleCache.Get(notifyRuleId)
+			if notifyRule == nil {
+				continue
+			}
+
+			if !notifyRule.Enable {
+				continue
+			}
+
+			var processors []models.Processor
+			for _, pipelineConfig := range notifyRule.PipelineConfigs {
+				if !pipelineConfig.Enable {
+					continue
+				}
+
+				eventPipeline := e.eventProcessorCache.Get(pipelineConfig.PipelineId)
+				if eventPipeline == nil {
+					logger.Warningf("notify_id: %d, event:%+v, processor not found", notifyRuleId, eventCopy)
+					continue
+				}
+
+				if !pipelineApplicable(eventPipeline, eventCopy) {
+					logger.Debugf("notify_id: %d, event:%+v, pipeline_id: %d, not applicable", notifyRuleId, eventCopy, pipelineConfig.PipelineId)
+					continue
+				}
+
+				processors = append(processors, e.eventProcessorCache.GetProcessorsById(pipelineConfig.PipelineId)...)
+			}
+
+			for _, processor := range processors {
+				logger.Infof("before processor notify_id: %d, event:%+v, processor:%+v", notifyRuleId, eventCopy, processor)
+				eventCopy, res, err := processor.Process(e.ctx, eventCopy)
+				logger.Infof("after processor notify_id: %d, event:%+v, processor:%+v, res:%v, err:%v", notifyRuleId, eventCopy, processor, res, err)
+				if eventCopy == nil {
+					logger.Warningf("notify_id: %d, event:%+v, processor:%+v, event is nil", notifyRuleId, eventCopy, processor)
+					break
+				}
+			}
+
+			if eventCopy == nil {
+				// 如果 eventCopy 为 nil，说明 eventCopy 被 processor drop 掉了, 不再发送通知
+				continue
+			}
+
+			// notify
+			for i := range notifyRule.NotifyConfigs {
+				if !NotifyRuleApplicable(&notifyRule.NotifyConfigs[i], eventCopy) {
+					continue
+				}
+				notifyChannel := e.notifyChannelCache.Get(notifyRule.NotifyConfigs[i].ChannelID)
+				messageTemplate := e.messageTemplateCache.Get(notifyRule.NotifyConfigs[i].TemplateID)
+				if notifyChannel == nil {
+					sender.NotifyRecord(e.ctx, []*models.AlertCurEvent{eventCopy}, notifyRuleId, fmt.Sprintf("notify_channel_id:%d", notifyRule.NotifyConfigs[i].ChannelID), "", "", errors.New("notify_channel not found"))
+					logger.Warningf("notify_id: %d, event:%+v, channel_id:%d, template_id: %d, notify_channel not found", notifyRuleId, eventCopy, notifyRule.NotifyConfigs[i].ChannelID, notifyRule.NotifyConfigs[i].TemplateID)
+					continue
+				}
+
+				if notifyChannel.RequestType != "flashduty" && messageTemplate == nil {
+					logger.Warningf("notify_id: %d, channel_name: %v, event:%+v, template_id: %d, message_template not found", notifyRuleId, notifyChannel.Ident, eventCopy, notifyRule.NotifyConfigs[i].TemplateID)
+					sender.NotifyRecord(e.ctx, []*models.AlertCurEvent{eventCopy}, notifyRuleId, notifyChannel.Name, "", "", errors.New("message_template not found"))
+
+					continue
+				}
+
+				// todo go send
+				// todo 聚合 event
+				go e.sendV2([]*models.AlertCurEvent{eventCopy}, notifyRuleId, &notifyRule.NotifyConfigs[i], notifyChannel, messageTemplate)
+			}
+		}
+	}
+}
+
+func pipelineApplicable(pipeline *models.EventPipeline, event *models.AlertCurEvent) bool {
+	if pipeline == nil {
+		return true
+	}
+
+	if !pipeline.FilterEnable {
+		return true
+	}
+
+	tagMatch := true
+	if len(pipeline.LabelFilters) > 0 {
+		for i := range pipeline.LabelFilters {
+			if pipeline.LabelFilters[i].Func == "" {
+				pipeline.LabelFilters[i].Func = pipeline.LabelFilters[i].Op
+			}
+		}
+
+		tagFilters, err := models.ParseTagFilter(pipeline.LabelFilters)
+		if err != nil {
+			logger.Errorf("pipeline applicable failed to parse tag filter: %v event:%+v pipeline:%+v", err, event, pipeline)
+			return false
+		}
+		tagMatch = common.MatchTags(event.TagsMap, tagFilters)
+	}
+
+	attributesMatch := true
+	if len(pipeline.AttrFilters) > 0 {
+		tagFilters, err := models.ParseTagFilter(pipeline.AttrFilters)
+		if err != nil {
+			logger.Errorf("pipeline applicable failed to parse tag filter: %v event:%+v pipeline:%+v err:%v", tagFilters, event, pipeline, err)
+			return false
+		}
+
+		attributesMatch = common.MatchTags(event.JsonTagsAndValue(), tagFilters)
+	}
+
+	return tagMatch && attributesMatch
+}
+
+func NotifyRuleApplicable(notifyConfig *models.NotifyConfig, event *models.AlertCurEvent) bool {
+	tm := time.Unix(event.TriggerTime, 0)
+	triggerTime := tm.Format("15:04")
+	triggerWeek := int(tm.Weekday())
+
+	timeMatch := false
+
+	if len(notifyConfig.TimeRanges) == 0 {
+		timeMatch = true
+	}
+	for j := range notifyConfig.TimeRanges {
+		if timeMatch {
+			break
+		}
+		enableStime := notifyConfig.TimeRanges[j].Start
+		enableEtime := notifyConfig.TimeRanges[j].End
+		enableDaysOfWeek := notifyConfig.TimeRanges[j].Week
+		length := len(enableDaysOfWeek)
+		// enableStime,enableEtime,enableDaysOfWeek三者长度肯定相同，这里循环一个即可
+		for i := 0; i < length; i++ {
+			if enableDaysOfWeek[i] != triggerWeek {
+				continue
+			}
+
+			if enableStime < enableEtime {
+				if enableEtime == "23:59" {
+					// 02:00-23:59，这种情况做个特殊处理，相当于左闭右闭区间了
+					if triggerTime < enableStime {
+						// mute, 即没生效
+						continue
+					}
+				} else {
+					// 02:00-04:00 或者 02:00-24:00
+					if triggerTime < enableStime || triggerTime >= enableEtime {
+						// mute, 即没生效
+						continue
+					}
+				}
+			} else if enableStime > enableEtime {
+				// 21:00-09:00
+				if triggerTime < enableStime && triggerTime >= enableEtime {
+					// mute, 即没生效
+					continue
+				}
+			}
+
+			// 到这里说明当前时刻在告警规则的某组生效时间范围内，即没有 mute，直接返回 false
+			timeMatch = true
+			break
+		}
+	}
+
+	severityMatch := false
+	for i := range notifyConfig.Severities {
+		if notifyConfig.Severities[i] == event.Severity {
+			severityMatch = true
+		}
+	}
+
+	tagMatch := true
+	if len(notifyConfig.LabelKeys) > 0 {
+		for i := range notifyConfig.LabelKeys {
+			if notifyConfig.LabelKeys[i].Func == "" {
+				notifyConfig.LabelKeys[i].Func = notifyConfig.LabelKeys[i].Op
+			}
+		}
+
+		tagFilters, err := models.ParseTagFilter(notifyConfig.LabelKeys)
+		if err != nil {
+			logger.Errorf("notify send failed to parse tag filter: %v event:%+v notify_config:%+v", err, event, notifyConfig)
+			return false
+		}
+		tagMatch = common.MatchTags(event.TagsMap, tagFilters)
+	}
+
+	attributesMatch := true
+	if len(notifyConfig.Attributes) > 0 {
+		tagFilters, err := models.ParseTagFilter(notifyConfig.Attributes)
+		if err != nil {
+			logger.Errorf("notify send failed to parse tag filter: %v event:%+v notify_config:%+v err:%v", tagFilters, event, notifyConfig, err)
+			return false
+		}
+
+		attributesMatch = common.MatchTags(event.JsonTagsAndValue(), tagFilters)
+	}
+	logger.Infof("notify send timeMatch:%v severityMatch:%v tagMatch:%v attributesMatch:%v event:%+v notify_config:%+v", timeMatch, severityMatch, tagMatch, attributesMatch, event, notifyConfig)
+	return timeMatch && severityMatch && tagMatch && attributesMatch
+}
+
+func GetNotifyConfigParams(notifyConfig *models.NotifyConfig, contactKey string, userCache *memsto.UserCacheType, userGroupCache *memsto.UserGroupCacheType) ([]string, []int64, map[string]string) {
+	customParams := make(map[string]string)
+	var flashDutyChannelIDs []int64
+	var userInfoParams models.CustomParams
+
+	for key, value := range notifyConfig.Params {
+		switch key {
+		case "user_ids", "user_group_ids", "ids":
+			if data, err := json.Marshal(value); err == nil {
+				var ids []int64
+				if json.Unmarshal(data, &ids) == nil {
+					if key == "user_ids" {
+						userInfoParams.UserIDs = ids
+					} else if key == "user_group_ids" {
+						userInfoParams.UserGroupIDs = ids
+					} else if key == "ids" {
+						flashDutyChannelIDs = ids
+					}
+				}
+			}
+		default:
+			customParams[key] = value.(string)
+		}
+	}
+
+	if len(userInfoParams.UserIDs) == 0 && len(userInfoParams.UserGroupIDs) == 0 {
+		return []string{}, flashDutyChannelIDs, customParams
+	}
+
+	userIds := make([]int64, 0)
+	userIds = append(userIds, userInfoParams.UserIDs...)
+
+	if len(userInfoParams.UserGroupIDs) > 0 {
+		userGroups := userGroupCache.GetByUserGroupIds(userInfoParams.UserGroupIDs)
+		for _, userGroup := range userGroups {
+			userIds = append(userIds, userGroup.UserIds...)
+		}
+	}
+
+	users := userCache.GetByUserIds(userIds)
+	visited := make(map[int64]bool)
+	sendtos := make([]string, 0)
+	for _, user := range users {
+		if visited[user.Id] {
+			continue
+		}
+		var sendto string
+		if contactKey == "phone" {
+			sendto = user.Phone
+		} else if contactKey == "email" {
+			sendto = user.Email
+		} else {
+			sendto, _ = user.ExtractToken(contactKey)
+		}
+
+		if sendto == "" {
+			continue
+		}
+		sendtos = append(sendtos, sendto)
+		visited[user.Id] = true
+	}
+
+	return sendtos, flashDutyChannelIDs, customParams
+}
+
+func (e *Dispatch) sendV2(events []*models.AlertCurEvent, notifyRuleId int64, notifyConfig *models.NotifyConfig, notifyChannel *models.NotifyChannelConfig, messageTemplate *models.MessageTemplate) {
+	if len(events) == 0 {
+		logger.Errorf("notify_id: %d events is empty", notifyRuleId)
+		return
+	}
+
+	tplContent := make(map[string]interface{})
+	if notifyChannel.RequestType != "flashduty" {
+		tplContent = messageTemplate.RenderEvent(events)
+	}
+
+	var contactKey string
+	if notifyChannel.ParamConfig != nil && notifyChannel.ParamConfig.UserInfo != nil {
+		contactKey = notifyChannel.ParamConfig.UserInfo.ContactKey
+	}
+
+	sendtos, flashDutyChannelIDs, customParams := GetNotifyConfigParams(notifyConfig, contactKey, e.userCache, e.userGroupCache)
+
+	e.Astats.GaugeNotifyRecordQueueSize.Inc()
+	defer e.Astats.GaugeNotifyRecordQueueSize.Dec()
+
+	switch notifyChannel.RequestType {
+	case "flashduty":
+		if len(flashDutyChannelIDs) == 0 {
+			flashDutyChannelIDs = []int64{0} // 如果 flashduty 通道没有配置，则使用 0, 给 SendFlashDuty 判断使用, 不给 flashduty 传 channel_id 参数
+		}
+
+		for i := range flashDutyChannelIDs {
+			respBody, err := notifyChannel.SendFlashDuty(events, flashDutyChannelIDs[i], e.notifyChannelCache.GetHttpClient(notifyChannel.ID))
+			logger.Infof("notify_id: %d, channel_name: %v, event:%+v, IntegrationUrl: %v dutychannel_id: %v, respBody: %v, err: %v", notifyRuleId, notifyChannel.Name, events[0], notifyChannel.RequestConfig.FlashDutyRequestConfig.IntegrationUrl, flashDutyChannelIDs[i], respBody, err)
+			sender.NotifyRecord(e.ctx, events, notifyRuleId, notifyChannel.Name, strconv.FormatInt(flashDutyChannelIDs[i], 10), respBody, err)
+		}
+		return
+	case "http":
+		if e.notifyChannelCache.HttpConcurrencyAdd(notifyChannel.ID) {
+			defer e.notifyChannelCache.HttpConcurrencyDone(notifyChannel.ID)
+		}
+		if notifyChannel.RequestConfig == nil {
+			logger.Warningf("notify_id: %d, channel_name: %v, event:%+v, request config not found", notifyRuleId, notifyChannel.Name, events[0])
+		}
+
+		if notifyChannel.RequestConfig.HTTPRequestConfig == nil {
+			logger.Warningf("notify_id: %d, channel_name: %v, event:%+v, http request config not found", notifyRuleId, notifyChannel.Name, events[0])
+		}
+
+		if NeedBatchContacts(notifyChannel.RequestConfig.HTTPRequestConfig) || len(sendtos) == 0 {
+			resp, err := notifyChannel.SendHTTP(events, tplContent, customParams, sendtos, e.notifyChannelCache.GetHttpClient(notifyChannel.ID))
+			logger.Infof("notify_id: %d, channel_name: %v, event:%+v, tplContent:%s, customParams:%v, userInfo:%+v, respBody: %v, err: %v", notifyRuleId, notifyChannel.Name, events[0], tplContent, customParams, sendtos, resp, err)
+
+			sender.NotifyRecord(e.ctx, events, notifyRuleId, notifyChannel.Name, getSendTarget(customParams, sendtos), resp, err)
+		} else {
+			for i := range sendtos {
+				resp, err := notifyChannel.SendHTTP(events, tplContent, customParams, []string{sendtos[i]}, e.notifyChannelCache.GetHttpClient(notifyChannel.ID))
+				logger.Infof("notify_id: %d, channel_name: %v, event:%+v, tplContent:%s, customParams:%v, userInfo:%+v, respBody: %v, err: %v", notifyRuleId, notifyChannel.Name, events[0], tplContent, customParams, sendtos[i], resp, err)
+				sender.NotifyRecord(e.ctx, events, notifyRuleId, notifyChannel.Name, getSendTarget(customParams, []string{sendtos[i]}), resp, err)
+			}
+		}
+
+	case "smtp":
+		notifyChannel.SendEmail(notifyRuleId, events, tplContent, sendtos, e.notifyChannelCache.GetSmtpClient(notifyChannel.ID))
+
+	case "script":
+		target, res, err := notifyChannel.SendScript(events, tplContent, customParams, sendtos)
+		logger.Infof("notify_id: %d, channel_name: %v, event:%+v, tplContent:%s, customParams:%v, target:%s, res:%s, err:%v", notifyRuleId, notifyChannel.Name, events[0], tplContent, customParams, target, res, err)
+		sender.NotifyRecord(e.ctx, events, notifyRuleId, notifyChannel.Name, target, res, err)
+	default:
+		logger.Warningf("notify_id: %d, channel_name: %v, event:%+v send type not found", notifyRuleId, notifyChannel.Name, events[0])
+	}
+}
+
+func NeedBatchContacts(requestConfig *models.HTTPRequestConfig) bool {
+	b, _ := json.Marshal(requestConfig)
+	return strings.Contains(string(b), "$sendtos")
+}
+
 // HandleEventNotify 处理event事件的主逻辑
 // event: 告警/恢复事件
 // isSubscribe: 告警事件是否由subscribe的配置产生
@@ -140,11 +503,6 @@ func (e *Dispatch) HandleEventNotify(event *models.AlertCurEvent, isSubscribe bo
 		return
 	}
 
-	if e.blockEventNotify(rule, event) {
-		logger.Infof("block event notify: rule_id:%d event:%+v", rule.Id, event)
-		return
-	}
-
 	fillUsers(event, e.userCache, e.userGroupCache)
 
 	var (
@@ -172,7 +530,7 @@ func (e *Dispatch) HandleEventNotify(event *models.AlertCurEvent, isSubscribe bo
 		notifyTarget.AndMerge(handler(rule, event, notifyTarget, e))
 	}
 
-	// 处理事件发送,这里用一个goroutine处理一个event的所有发送事件
+	go e.HandleEventWithNotifyRule(event)
 	go e.Send(rule, event, notifyTarget, isSubscribe)
 
 	// 如果是不是订阅规则出现的event, 则需要处理订阅规则的event
@@ -181,25 +539,6 @@ func (e *Dispatch) HandleEventNotify(event *models.AlertCurEvent, isSubscribe bo
 	}
 }
 
-func (e *Dispatch) blockEventNotify(rule *models.AlertRule, event *models.AlertCurEvent) bool {
-	ruleType := rule.GetRuleType()
-
-	// 若为机器则先看机器是否删除
-	if ruleType == models.HOST {
-		host, ok := e.targetCache.Get(event.TagsMap["ident"])
-		if !ok || host == nil {
-			return true
-		}
-	}
-
-	// 恢复通知，检测规则配置是否改变
-	// if event.IsRecovered && event.RuleHash != rule.Hash() {
-	// 	return true
-	// }
-
-	return false
-}
-
 func (e *Dispatch) handleSubs(event *models.AlertCurEvent) {
 	// handle alert subscribes
 	subscribes := make([]*models.AlertSubscribe, 0)
@@ -369,6 +708,11 @@ func (e *Dispatch) HandleIbex(rule *models.AlertRule, event *models.AlertCurEven
 	}
 	json.Unmarshal([]byte(rule.RuleConfig), &ruleConfig)
 
+	if event.IsRecovered {
+		// 恢复事件不需要走故障自愈的逻辑
+		return
+	}
+
 	for _, t := range ruleConfig.TaskTpls {
 		if t.TplId == 0 {
 			continue
@@ -447,3 +791,22 @@ func mapKeys(m map[int64]struct{}) []int64 {
 	}
 	return lst
 }
+
+func getSendTarget(customParams map[string]string, sendtos []string) string {
+	if len(customParams) == 0 {
+		return strings.Join(sendtos, ",")
+	}
+
+	values := make([]string, 0)
+	for _, value := range customParams {
+		runes := []rune(value)
+		if len(runes) <= 4 {
+			values = append(values, value)
+		} else {
+			maskedValue := string(runes[:len(runes)-4]) + "****"
+			values = append(values, maskedValue)
+		}
+	}
+
+	return strings.Join(values, ",")
+}

alert/dispatch/log.go

@@ -18,12 +18,13 @@ func LogEvent(event *models.AlertCurEvent, location string, err ...error) {
 	}
 
 	logger.Infof(
-		"event(%s %s) %s: rule_id=%d sub_id:%d cluster:%s %v%s@%d %s",
+		"event(%s %s) %s: rule_id=%d sub_id:%d notify_rule_ids:%v cluster:%s %v%s@%d %s",
 		event.Hash,
 		status,
 		location,
 		event.RuleId,
 		event.SubRuleId,
+		event.NotifyRuleIds,
 		event.Cluster,
 		event.TagsJSON,
 		event.TriggerValue,

alert/eval/alert_rule.go

@@ -93,7 +93,7 @@ func (s *Scheduler) syncAlertRules() {
 		}
 
 		ruleType := rule.GetRuleType()
-		if rule.IsPrometheusRule() || rule.IsLokiRule() || rule.IsTdengineRule() || rule.IsClickHouseRule() || rule.IsElasticSearch() {
+		if rule.IsPrometheusRule() || rule.IsInnerRule() {
 			datasourceIds := s.datasourceCache.GetIDsByDsCateAndQueries(rule.Cate, rule.DatasourceQueries)
 			for _, dsId := range datasourceIds {
 				if !naming.DatasourceHashRing.IsHit(strconv.FormatInt(dsId, 10), fmt.Sprintf("%d", rule.Id), s.aconf.Heartbeat.Endpoint) {

alert/eval/eval.go

@@ -13,6 +13,7 @@ import (
 	"sync"
 	"time"
 
+	"github.com/ccfos/nightingale/v6/alert/astats"
 	"github.com/ccfos/nightingale/v6/alert/common"
 	"github.com/ccfos/nightingale/v6/alert/process"
 	"github.com/ccfos/nightingale/v6/dscache"
@@ -20,6 +21,7 @@ import (
 	"github.com/ccfos/nightingale/v6/pkg/ctx"
 	"github.com/ccfos/nightingale/v6/pkg/hash"
 	"github.com/ccfos/nightingale/v6/pkg/parser"
+	"github.com/ccfos/nightingale/v6/pkg/poster"
 	promsdk "github.com/ccfos/nightingale/v6/pkg/prom"
 	promql2 "github.com/ccfos/nightingale/v6/pkg/promql"
 	"github.com/ccfos/nightingale/v6/pkg/unit"
@@ -35,7 +37,6 @@ type AlertRuleWorker struct {
 	DatasourceId int64
 	Quit         chan struct{}
 	Inhibit      bool
-	Severity     int
 
 	Rule *models.AlertRule
 
@@ -98,7 +99,7 @@ func NewAlertRuleWorker(rule *models.AlertRule, datasourceId int64, Processor *p
 		rule.CronPattern = fmt.Sprintf("@every %ds", interval)
 	}
 
-	arw.Scheduler = cron.New(cron.WithSeconds())
+	arw.Scheduler = cron.New(cron.WithSeconds(), cron.WithChain(cron.SkipIfStillRunning(cron.DefaultLogger)))
 
 	entryID, err := arw.Scheduler.AddFunc(rule.CronPattern, func() {
 		arw.Eval()
@@ -171,7 +172,7 @@ func (arw *AlertRuleWorker) Eval() {
 	case models.LOKI:
 		anomalyPoints, err = arw.GetPromAnomalyPoint(cachedRule.RuleConfig)
 	default:
-		anomalyPoints, recoverPoints = arw.GetAnomalyPoint(cachedRule, arw.Processor.DatasourceId())
+		anomalyPoints, recoverPoints, err = arw.GetAnomalyPoint(cachedRule, arw.Processor.DatasourceId())
 	}
 
 	if err != nil {
@@ -231,32 +232,46 @@ func (arw *AlertRuleWorker) Stop() {
 
 func (arw *AlertRuleWorker) GetPromAnomalyPoint(ruleConfig string) ([]models.AnomalyPoint, error) {
 	var lst []models.AnomalyPoint
-	var severity int
+	start := time.Now()
+	defer func() {
+		arw.Processor.Stats.GaugeRuleEvalDuration.WithLabelValues(fmt.Sprintf("%v", arw.Rule.Id), fmt.Sprintf("%v", arw.Processor.DatasourceId())).Set(float64(time.Since(start).Milliseconds()))
+	}()
 
 	var rule *models.PromRuleConfig
 	if err := json.Unmarshal([]byte(ruleConfig), &rule); err != nil {
 		logger.Errorf("rule_eval:%s rule_config:%s, error:%v", arw.Key(), ruleConfig, err)
 		arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), GET_RULE_CONFIG, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
+		arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+			fmt.Sprintf("%v", arw.Rule.Id),
+			fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+			"",
+		).Set(0)
 		return lst, err
 	}
 
 	if rule == nil {
 		logger.Errorf("rule_eval:%s rule_config:%s, error:rule is nil", arw.Key(), ruleConfig)
 		arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), GET_RULE_CONFIG, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
+		arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+			fmt.Sprintf("%v", arw.Rule.Id),
+			fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+			"",
+		).Set(0)
 		return lst, errors.New("rule is nil")
 	}
 
 	arw.Inhibit = rule.Inhibit
-	for _, query := range rule.Queries {
-		if query.Severity < severity {
-			arw.Severity = query.Severity
-		}
-
+	for i, query := range rule.Queries {
 		readerClient := arw.PromClients.GetCli(arw.DatasourceId)
 
 		if readerClient == nil {
 			logger.Warningf("rule_eval:%s error reader client is nil", arw.Key())
 			arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), GET_CLIENT, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
+			arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+				fmt.Sprintf("%v", arw.Rule.Id),
+				fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+				fmt.Sprintf("%v", i),
+			).Set(-2)
 			continue
 		}
 
@@ -265,9 +280,21 @@ func (arw *AlertRuleWorker) GetPromAnomalyPoint(ruleConfig string) ([]models.Ano
 			if hasLabelLossAggregator(query) || notExactMatch(query) {
 				// 若有聚合函数或非精确匹配则需要先填充变量然后查询，这个方式效率较低
 				anomalyPoints = arw.VarFillingBeforeQuery(query, readerClient)
+				arw.Processor.Stats.CounterVarFillingQuery.WithLabelValues(
+					fmt.Sprintf("%v", arw.Rule.Id),
+					fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+					fmt.Sprintf("%v", i),
+					"BeforeQuery",
+				).Inc()
 			} else {
 				// 先查询再过滤变量，效率较高，但无法处理有聚合函数的情况
 				anomalyPoints = arw.VarFillingAfterQuery(query, readerClient)
+				arw.Processor.Stats.CounterVarFillingQuery.WithLabelValues(
+					fmt.Sprintf("%v", arw.Rule.Id),
+					fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+					fmt.Sprintf("%v", i),
+					"AfterQuery",
+				).Inc()
 			}
 			lst = append(lst, anomalyPoints...)
 		} else {
@@ -286,12 +313,17 @@ func (arw *AlertRuleWorker) GetPromAnomalyPoint(ruleConfig string) ([]models.Ano
 			}
 
 			var warnings promsdk.Warnings
-			arw.Processor.Stats.CounterQueryDataTotal.WithLabelValues(fmt.Sprintf("%d", arw.DatasourceId)).Inc()
+			arw.Processor.Stats.CounterQueryDataTotal.WithLabelValues(fmt.Sprintf("%d", arw.DatasourceId), fmt.Sprintf("%d", arw.Rule.Id)).Inc()
 			value, warnings, err := readerClient.Query(context.Background(), promql, time.Now())
 			if err != nil {
 				logger.Errorf("rule_eval:%s promql:%s, error:%v", arw.Key(), promql, err)
 				arw.Processor.Stats.CounterQueryDataErrorTotal.WithLabelValues(fmt.Sprintf("%d", arw.DatasourceId)).Inc()
 				arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), QUERY_DATA, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
+				arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+					fmt.Sprintf("%v", arw.Rule.Id),
+					fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+					fmt.Sprintf("%v", i),
+				).Set(-1)
 				return lst, err
 			}
 
@@ -301,8 +333,14 @@ func (arw *AlertRuleWorker) GetPromAnomalyPoint(ruleConfig string) ([]models.Ano
 				arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), QUERY_DATA, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
 			}
 
-			logger.Debugf("rule_eval:%s query:%+v, value:%v", arw.Key(), query, value)
+			logger.Infof("rule_eval:%s query:%+v, value:%v", arw.Key(), query, value)
 			points := models.ConvertAnomalyPoints(value)
+			arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+				fmt.Sprintf("%v", arw.Rule.Id),
+				fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+				fmt.Sprintf("%v", i),
+			).Set(float64(len(points)))
+
 			for i := 0; i < len(points); i++ {
 				points[i].Severity = query.Severity
 				points[i].Query = promql
@@ -313,7 +351,14 @@ func (arw *AlertRuleWorker) GetPromAnomalyPoint(ruleConfig string) ([]models.Ano
 
 			lst = append(lst, points...)
 		}
+
+		arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+			fmt.Sprintf("%v", arw.Rule.Id),
+			fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+			fmt.Sprintf("%v", i),
+		).Set(float64(len(lst)))
 	}
+
 	return lst, nil
 }
 
@@ -379,6 +424,7 @@ func (arw *AlertRuleWorker) VarFillingAfterQuery(query models.PromQuery, readerC
 				realQuery = strings.Replace(realQuery, fmt.Sprintf("$%s", key), val, -1)
 			}
 			// 得到满足值变量的所有结果
+			arw.Processor.Stats.CounterQueryDataTotal.WithLabelValues(fmt.Sprintf("%d", arw.DatasourceId), fmt.Sprintf("%d", arw.Rule.Id)).Inc()
 			value, _, err := readerClient.Query(context.Background(), curQuery, time.Now())
 			if err != nil {
 				logger.Errorf("rule_eval:%s, promql:%s, error:%v", arw.Key(), curQuery, err)
@@ -540,7 +586,7 @@ func (arw *AlertRuleWorker) getParamPermutation(paramVal map[string]models.Param
 				logger.Errorf("query:%s fail to unmarshalling into string slice, error:%v", paramQuery.Query, err)
 			}
 			if len(query) == 0 {
-				paramsKeyAllLabel, err := getParamKeyAllLabel(varToLabel[paramKey], originPromql, readerClient)
+				paramsKeyAllLabel, err := getParamKeyAllLabel(varToLabel[paramKey], originPromql, readerClient, arw.DatasourceId, arw.Rule.Id, arw.Processor.Stats)
 				if err != nil {
 					logger.Errorf("rule_eval:%s, fail to getParamKeyAllLabel, error:%v query:%s", arw.Key(), err, paramQuery.Query)
 				}
@@ -571,7 +617,7 @@ func (arw *AlertRuleWorker) getParamPermutation(paramVal map[string]models.Param
 	return res, nil
 }
 
-func getParamKeyAllLabel(paramKey string, promql string, client promsdk.API) ([]string, error) {
+func getParamKeyAllLabel(paramKey string, promql string, client promsdk.API, dsId int64, rid int64, stats *astats.Stats) ([]string, error) {
 	labels, metricName, err := promql2.GetLabelsAndMetricNameWithReplace(promql, "$")
 	if err != nil {
 		return nil, fmt.Errorf("promql:%s, get labels error:%v", promql, err)
@@ -585,6 +631,7 @@ func getParamKeyAllLabel(paramKey string, promql string, client promsdk.API) ([]
 	}
 	pr := metricName + "{" + strings.Join(labelstrs, ",") + "}"
 
+	stats.CounterQueryDataTotal.WithLabelValues(fmt.Sprintf("%d", dsId), fmt.Sprintf("%d", rid)).Inc()
 	value, _, err := client.Query(context.Background(), pr, time.Now())
 	if err != nil {
 		return nil, fmt.Errorf("promql: %s query error: %v", pr, err)
@@ -639,16 +686,27 @@ func (arw *AlertRuleWorker) getHostIdents(paramQuery models.ParamQuery) ([]strin
 		return nil, err
 	}
 
-	hostsQuery := models.GetHostsQuery(queries)
-	session := models.TargetFilterQueryBuild(arw.Ctx, hostsQuery, 0, 0)
-	var lst []*models.Target
-	err = session.Find(&lst).Error
-	if err != nil {
-		return nil, err
-	}
-	for i := range lst {
-		params = append(params, lst[i].Ident)
+	if !arw.Ctx.IsCenter {
+		lst, err := poster.PostByUrlsWithResp[[]*models.Target](arw.Ctx, "/v1/n9e/targets-of-host-query", queries)
+		if err != nil {
+			return nil, err
+		}
+		for i := range lst {
+			params = append(params, lst[i].Ident)
+		}
+	} else {
+		hostsQuery := models.GetHostsQuery(queries)
+		session := models.TargetFilterQueryBuild(arw.Ctx, hostsQuery, 0, 0)
+		var lst []*models.Target
+		err = session.Find(&lst).Error
+		if err != nil {
+			return nil, err
+		}
+		for i := range lst {
+			params = append(params, lst[i].Ident)
+		}
 	}
+
 	arw.HostAndDeviceIdentCache.Store(cacheKey, params)
 	return params, nil
 }
@@ -688,28 +746,37 @@ func combine(paramKeys []string, paraMap map[string][]string, index int, current
 
 func (arw *AlertRuleWorker) GetHostAnomalyPoint(ruleConfig string) ([]models.AnomalyPoint, error) {
 	var lst []models.AnomalyPoint
-	var severity int
+	start := time.Now()
+	defer func() {
+		arw.Processor.Stats.GaugeRuleEvalDuration.WithLabelValues(fmt.Sprintf("%v", arw.Rule.Id), fmt.Sprintf("%v", arw.Processor.DatasourceId())).Set(float64(time.Since(start).Milliseconds()))
+	}()
 
 	var rule *models.HostRuleConfig
 	if err := json.Unmarshal([]byte(ruleConfig), &rule); err != nil {
 		logger.Errorf("rule_eval:%s rule_config:%s, error:%v", arw.Key(), ruleConfig, err)
 		arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), GET_RULE_CONFIG, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
+		arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+			fmt.Sprintf("%v", arw.Rule.Id),
+			fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+			"",
+		).Set(0)
 		return lst, err
 	}
 
 	if rule == nil {
 		logger.Errorf("rule_eval:%s rule_config:%s, error:rule is nil", arw.Key(), ruleConfig)
 		arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), GET_RULE_CONFIG, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
+		arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+			fmt.Sprintf("%v", arw.Rule.Id),
+			fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+			"",
+		).Set(0)
 		return lst, errors.New("rule is nil")
 	}
 
 	arw.Inhibit = rule.Inhibit
 	now := time.Now().Unix()
 	for _, trigger := range rule.Triggers {
-		if trigger.Severity < severity {
-			arw.Severity = trigger.Severity
-		}
-
 		switch trigger.Type {
 		case "target_miss":
 			t := now - int64(trigger.Duration)
@@ -734,6 +801,11 @@ func (arw *AlertRuleWorker) GetHostAnomalyPoint(ruleConfig string) ([]models.Ano
 			idents = append(idents, engineIdents...)
 
 			if len(idents) == 0 {
+				arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+					fmt.Sprintf("%v", arw.Rule.Id),
+					fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+					"",
+				).Set(0)
 				continue
 			}
 
@@ -744,6 +816,12 @@ func (arw *AlertRuleWorker) GetHostAnomalyPoint(ruleConfig string) ([]models.Ano
 					missTargets = append(missTargets, ident)
 				}
 			}
+			arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+				fmt.Sprintf("%v", arw.Rule.Id),
+				fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+				"",
+			).Set(float64(len(missTargets)))
+
 			logger.Debugf("rule_eval:%s missTargets:%v", arw.Key(), missTargets)
 			targets := arw.Processor.TargetCache.Gets(missTargets)
 			for _, target := range targets {
@@ -758,6 +836,11 @@ func (arw *AlertRuleWorker) GetHostAnomalyPoint(ruleConfig string) ([]models.Ano
 		case "offset":
 			idents, exists := arw.Processor.TargetsOfAlertRuleCache.Get(arw.Processor.EngineName, arw.Rule.Id)
 			if !exists {
+				arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+					fmt.Sprintf("%v", arw.Rule.Id),
+					fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+					"",
+				).Set(0)
 				logger.Warningf("rule_eval:%s targets not found", arw.Key())
 				arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), QUERY_DATA, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
 				continue
@@ -790,6 +873,11 @@ func (arw *AlertRuleWorker) GetHostAnomalyPoint(ruleConfig string) ([]models.Ano
 			}
 
 			logger.Debugf("rule_eval:%s offsetIdents:%v", arw.Key(), offsetIdents)
+			arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+				fmt.Sprintf("%v", arw.Rule.Id),
+				fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+				"",
+			).Set(float64(len(offsetIdents)))
 			for host, offset := range offsetIdents {
 				m := make(map[string]string)
 				target, exists := arw.Processor.TargetCache.Get(host)
@@ -806,6 +894,11 @@ func (arw *AlertRuleWorker) GetHostAnomalyPoint(ruleConfig string) ([]models.Ano
 			t := now - int64(trigger.Duration)
 			idents, exists := arw.Processor.TargetsOfAlertRuleCache.Get(arw.Processor.EngineName, arw.Rule.Id)
 			if !exists {
+				arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+					fmt.Sprintf("%v", arw.Rule.Id),
+					fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+					"",
+				).Set(0)
 				logger.Warningf("rule_eval:%s targets not found", arw.Key())
 				arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), QUERY_DATA, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
 				continue
@@ -819,6 +912,11 @@ func (arw *AlertRuleWorker) GetHostAnomalyPoint(ruleConfig string) ([]models.Ano
 				}
 			}
 			logger.Debugf("rule_eval:%s missTargets:%v", arw.Key(), missTargets)
+			arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+				fmt.Sprintf("%v", arw.Rule.Id),
+				fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+				"",
+			).Set(float64(len(missTargets)))
 			pct := float64(len(missTargets)) / float64(len(idents)) * 100
 			if pct >= float64(trigger.Percent) {
 				lst = append(lst, models.NewAnomalyPoint(trigger.Type, nil, now, pct, trigger.Severity))
@@ -1190,6 +1288,7 @@ func (arw *AlertRuleWorker) VarFillingBeforeQuery(query models.PromQuery, reader
 						<-semaphore
 						wg.Done()
 					}()
+					arw.Processor.Stats.CounterQueryDataTotal.WithLabelValues(fmt.Sprintf("%d", arw.DatasourceId), fmt.Sprintf("%d", arw.Rule.Id)).Inc()
 					value, _, err := readerClient.Query(context.Background(), promql, time.Now())
 					if err != nil {
 						logger.Errorf("rule_eval:%s, promql:%s, error:%v", arw.Key(), promql, err)
@@ -1323,15 +1422,26 @@ func fillVar(curRealQuery string, paramKey string, val string) string {
 	return curRealQuery
 }
 
-func (arw *AlertRuleWorker) GetAnomalyPoint(rule *models.AlertRule, dsId int64) ([]models.AnomalyPoint, []models.AnomalyPoint) {
+func (arw *AlertRuleWorker) GetAnomalyPoint(rule *models.AlertRule, dsId int64) ([]models.AnomalyPoint, []models.AnomalyPoint, error) {
 	// 获取查询和规则判断条件
+	start := time.Now()
+	defer func() {
+		arw.Processor.Stats.GaugeRuleEvalDuration.WithLabelValues(fmt.Sprintf("%v", arw.Rule.Id), fmt.Sprintf("%v", arw.Processor.DatasourceId())).Set(float64(time.Since(start).Milliseconds()))
+	}()
+
 	points := []models.AnomalyPoint{}
 	recoverPoints := []models.AnomalyPoint{}
 	ruleConfig := strings.TrimSpace(rule.RuleConfig)
 	if ruleConfig == "" {
-		logger.Warningf("rule_eval:%d promql is blank", rule.Id)
+		logger.Warningf("rule_eval:%d ruleConfig is blank", rule.Id)
 		arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), GET_RULE_CONFIG, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
-		return points, recoverPoints
+		arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+			fmt.Sprintf("%v", arw.Rule.Id),
+			fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+			"",
+		).Set(0)
+
+		return points, recoverPoints, fmt.Errorf("rule_eval:%d ruleConfig is blank", rule.Id)
 	}
 
 	var ruleQuery models.RuleQuery
@@ -1339,32 +1449,51 @@ func (arw *AlertRuleWorker) GetAnomalyPoint(rule *models.AlertRule, dsId int64)
 	if err != nil {
 		logger.Warningf("rule_eval:%d promql parse error:%s", rule.Id, err.Error())
 		arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), GET_RULE_CONFIG, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
-		return points, recoverPoints
+		return points, recoverPoints, fmt.Errorf("rule_eval:%d promql parse error:%s", rule.Id, err.Error())
 	}
 
 	arw.Inhibit = ruleQuery.Inhibit
 	if len(ruleQuery.Queries) > 0 {
 		seriesStore := make(map[uint64]models.DataResp)
 		seriesTagIndexes := make(map[string]map[uint64][]uint64, 0)
-		for _, query := range ruleQuery.Queries {
+		for i, query := range ruleQuery.Queries {
 			seriesTagIndex := make(map[uint64][]uint64)
 
 			plug, exists := dscache.DsCache.Get(rule.Cate, dsId)
 			if !exists {
 				logger.Warningf("rule_eval rid:%d datasource:%d not exists", rule.Id, dsId)
 				arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), GET_CLIENT, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
-				continue
+
+				arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+					fmt.Sprintf("%v", arw.Rule.Id),
+					fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+					fmt.Sprintf("%v", i),
+				).Set(-2)
+
+				return points, recoverPoints, fmt.Errorf("rule_eval:%d datasource:%d not exists", rule.Id, dsId)
 			}
 
 			ctx := context.WithValue(context.Background(), "delay", int64(rule.Delay))
 			series, err := plug.QueryData(ctx, query)
-			arw.Processor.Stats.CounterQueryDataTotal.WithLabelValues(fmt.Sprintf("%d", arw.DatasourceId)).Inc()
+			arw.Processor.Stats.CounterQueryDataTotal.WithLabelValues(fmt.Sprintf("%d", arw.DatasourceId), fmt.Sprintf("%d", rule.Id)).Inc()
 			if err != nil {
 				logger.Warningf("rule_eval rid:%d query data error: %v", rule.Id, err)
 				arw.Processor.Stats.CounterRuleEvalErrorTotal.WithLabelValues(fmt.Sprintf("%v", arw.Processor.DatasourceId()), GET_CLIENT, arw.Processor.BusiGroupCache.GetNameByBusiGroupId(arw.Rule.GroupId), fmt.Sprintf("%v", arw.Rule.Id)).Inc()
-				continue
+				arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+					fmt.Sprintf("%v", arw.Rule.Id),
+					fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+					fmt.Sprintf("%v", i),
+				).Set(-1)
+
+				return points, recoverPoints, fmt.Errorf("rule_eval:%d query data error: %v", rule.Id, err)
 			}
 
+			arw.Processor.Stats.GaugeQuerySeriesCount.WithLabelValues(
+				fmt.Sprintf("%v", arw.Rule.Id),
+				fmt.Sprintf("%v", arw.Processor.DatasourceId()),
+				fmt.Sprintf("%v", i),
+			).Set(float64(len(series)))
+
 			//  此条日志很重要，是告警判断的现场值
 			logger.Infof("rule_eval rid:%d req:%+v resp:%v", rule.Id, query, series)
 			for i := 0; i < len(series); i++ {
@@ -1390,6 +1519,7 @@ func (arw *AlertRuleWorker) GetAnomalyPoint(rule *models.AlertRule, dsId int64)
 		for _, query := range ruleQuery.Queries {
 			ref, unit, err := GetQueryRefAndUnit(query)
 			if err != nil {
+				logger.Warningf("rule_eval rid:%d query:%+v get ref and unit error:%s", rule.Id, query, err.Error())
 				continue
 			}
 			unitMap[ref] = unit
@@ -1469,6 +1599,11 @@ func (arw *AlertRuleWorker) GetAnomalyPoint(rule *models.AlertRule, dsId int64)
 						}
 					}
 
+					queries := ruleQuery.Queries
+					if sample.Query != "" {
+						queries = []interface{}{sample.Query}
+					}
+
 					point := models.AnomalyPoint{
 						Key:           sample.MetricName(),
 						Labels:        sample.Metric,
@@ -1477,7 +1612,7 @@ func (arw *AlertRuleWorker) GetAnomalyPoint(rule *models.AlertRule, dsId int64)
 						Values:        values,
 						Severity:      trigger.Severity,
 						Triggered:     isTriggered,
-						Query:         fmt.Sprintf("query:%+v trigger:%+v", ruleQuery.Queries, trigger),
+						Query:         fmt.Sprintf("query:%+v trigger:%+v", queries, trigger),
 						RecoverConfig: trigger.RecoverConfig,
 						ValuesUnit:    valuesUnitMap,
 					}
@@ -1551,5 +1686,5 @@ func (arw *AlertRuleWorker) GetAnomalyPoint(rule *models.AlertRule, dsId int64)
 		}
 	}
 
-	return points, recoverPoints
+	return points, recoverPoints, nil
 }

alert/mute/mute.go

@@ -12,28 +12,29 @@ import (
 	"github.com/toolkits/pkg/logger"
 )
 
-func IsMuted(rule *models.AlertRule, event *models.AlertCurEvent, targetCache *memsto.TargetCacheType, alertMuteCache *memsto.AlertMuteCacheType) (bool, string) {
+func IsMuted(rule *models.AlertRule, event *models.AlertCurEvent, targetCache *memsto.TargetCacheType, alertMuteCache *memsto.AlertMuteCacheType) (bool, string, int64) {
 	if rule.Disabled == 1 {
-		return true, "rule disabled"
+		return true, "rule disabled", 0
 	}
 
 	if TimeSpanMuteStrategy(rule, event) {
-		return true, "rule is not effective for period of time"
+		return true, "rule is not effective for period of time", 0
 	}
 
 	if IdentNotExistsMuteStrategy(rule, event, targetCache) {
-		return true, "ident not exists mute"
+		return true, "ident not exists mute", 0
 	}
 
 	if BgNotMatchMuteStrategy(rule, event, targetCache) {
-		return true, "bg not match mute"
+		return true, "bg not match mute", 0
 	}
 
-	if EventMuteStrategy(event, alertMuteCache) {
-		return true, "match mute rule"
+	hit, muteId := EventMuteStrategy(event, alertMuteCache)
+	if hit {
+		return true, "match mute rule", muteId
 	}
 
-	return false, ""
+	return false, "", 0
 }
 
 // TimeSpanMuteStrategy 根据规则配置的告警生效时间段过滤,如果产生的告警不在规则配置的告警生效时间段内,则不告警,即被mute
@@ -43,6 +44,12 @@ func TimeSpanMuteStrategy(rule *models.AlertRule, event *models.AlertCurEvent) b
 	triggerTime := tm.Format("15:04")
 	triggerWeek := strconv.Itoa(int(tm.Weekday()))
 
+	if rule.EnableDaysOfWeek == "" {
+		// 如果规则没有配置生效时间，则默认全天生效
+
+		return false
+	}
+
 	enableStime := strings.Fields(rule.EnableStime)
 	enableEtime := strings.Fields(rule.EnableEtime)
 	enableDaysOfWeek := strings.Split(rule.EnableDaysOfWeek, ";")
@@ -121,30 +128,26 @@ func BgNotMatchMuteStrategy(rule *models.AlertRule, event *models.AlertCurEvent,
 	return false
 }
 
-func EventMuteStrategy(event *models.AlertCurEvent, alertMuteCache *memsto.AlertMuteCacheType) bool {
+func EventMuteStrategy(event *models.AlertCurEvent, alertMuteCache *memsto.AlertMuteCacheType) (bool, int64) {
 	mutes, has := alertMuteCache.Gets(event.GroupId)
 	if !has || len(mutes) == 0 {
-		return false
+		return false, 0
 	}
 
 	for i := 0; i < len(mutes); i++ {
-		if matchMute(event, mutes[i]) {
-			return true
+		if MatchMute(event, mutes[i]) {
+			return true, mutes[i].Id
 		}
 	}
 
-	return false
+	return false, 0
 }
 
-// matchMute 如果传入了clock这个可选参数，就表示使用这个clock表示的时间，否则就从event的字段中取TriggerTime
-func matchMute(event *models.AlertCurEvent, mute *models.AlertMute, clock ...int64) bool {
+// MatchMute 如果传入了clock这个可选参数，就表示使用这个clock表示的时间，否则就从event的字段中取TriggerTime
+func MatchMute(event *models.AlertCurEvent, mute *models.AlertMute, clock ...int64) bool {
 	if mute.Disabled == 1 {
 		return false
 	}
-	ts := event.TriggerTime
-	if len(clock) > 0 {
-		ts = clock[0]
-	}
 
 	// 如果不是全局的，判断 匹配的 datasource id
 	if len(mute.DatasourceIdsJson) != 0 && mute.DatasourceIdsJson[0] != 0 && event.DatasourceId != 0 {
@@ -159,37 +162,21 @@ func matchMute(event *models.AlertCurEvent, mute *models.AlertMute, clock ...int
 		}
 	}
 
-	var matchTime bool
 	if mute.MuteTimeType == models.TimeRange {
-		if ts < mute.Btime || ts > mute.Etime {
+		if !mute.IsWithinTimeRange(event.TriggerTime) {
 			return false
 		}
-		matchTime = true
 	} else if mute.MuteTimeType == models.Periodic {
-		tm := time.Unix(event.TriggerTime, 0)
-		triggerTime := tm.Format("15:04")
-		triggerWeek := strconv.Itoa(int(tm.Weekday()))
-
-		for i := 0; i < len(mute.PeriodicMutesJson); i++ {
-			if strings.Contains(mute.PeriodicMutesJson[i].EnableDaysOfWeek, triggerWeek) {
-				if mute.PeriodicMutesJson[i].EnableStime == mute.PeriodicMutesJson[i].EnableEtime || (mute.PeriodicMutesJson[i].EnableStime == "00:00" && mute.PeriodicMutesJson[i].EnableEtime == "23:59") {
-					matchTime = true
-					break
-				} else if mute.PeriodicMutesJson[i].EnableStime < mute.PeriodicMutesJson[i].EnableEtime {
-					if triggerTime >= mute.PeriodicMutesJson[i].EnableStime && triggerTime < mute.PeriodicMutesJson[i].EnableEtime {
-						matchTime = true
-						break
-					}
-				} else {
-					if triggerTime >= mute.PeriodicMutesJson[i].EnableStime || triggerTime < mute.PeriodicMutesJson[i].EnableEtime {
-						matchTime = true
-						break
-					}
-				}
-			}
+		ts := event.TriggerTime
+		if len(clock) > 0 {
+			ts = clock[0]
 		}
-	}
-	if !matchTime {
+
+		if !mute.IsWithinPeriodicMute(ts) {
+			return false
+		}
+	} else {
+		logger.Warningf("mute time type invalid, %d", mute.MuteTimeType)
 		return false
 	}
 

alert/pipeline/pipeline.go

@@ -0,0 +1,12 @@
+package pipeline
+
+import (
+	_ "github.com/ccfos/nightingale/v6/alert/pipeline/processor/aisummary"
+	_ "github.com/ccfos/nightingale/v6/alert/pipeline/processor/callback"
+	_ "github.com/ccfos/nightingale/v6/alert/pipeline/processor/eventdrop"
+	_ "github.com/ccfos/nightingale/v6/alert/pipeline/processor/eventupdate"
+	_ "github.com/ccfos/nightingale/v6/alert/pipeline/processor/relabel"
+)
+
+func Init() {
+}

alert/pipeline/processor/aisummary/ai_summary.go

@@ -0,0 +1,198 @@
+package aisummary
+
+import (
+	"bytes"
+	"crypto/tls"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"strings"
+	"text/template"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/alert/pipeline/processor/callback"
+	"github.com/ccfos/nightingale/v6/alert/pipeline/processor/common"
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/ctx"
+	"github.com/ccfos/nightingale/v6/pkg/tplx"
+)
+
+const (
+	HTTP_STATUS_SUCCESS_MAX = 299
+)
+
+// AISummaryConfig 配置结构体
+type AISummaryConfig struct {
+	callback.HTTPConfig
+	ModelName      string                 `json:"model_name"`
+	APIKey         string                 `json:"api_key"`
+	PromptTemplate string                 `json:"prompt_template"`
+	CustomParams   map[string]interface{} `json:"custom_params"`
+}
+
+type Message struct {
+	Role    string `json:"role"`
+	Content string `json:"content"`
+}
+
+type ChatCompletionResponse struct {
+	Choices []struct {
+		Message struct {
+			Content string `json:"content"`
+		} `json:"message"`
+	} `json:"choices"`
+}
+
+func init() {
+	models.RegisterProcessor("ai_summary", &AISummaryConfig{})
+}
+
+func (c *AISummaryConfig) Init(settings interface{}) (models.Processor, error) {
+	result, err := common.InitProcessor[*AISummaryConfig](settings)
+	return result, err
+}
+
+func (c *AISummaryConfig) Process(ctx *ctx.Context, event *models.AlertCurEvent) (*models.AlertCurEvent, string, error) {
+	if c.Client == nil {
+		if err := c.initHTTPClient(); err != nil {
+			return event, "", fmt.Errorf("failed to initialize HTTP client: %v processor: %v", err, c)
+		}
+	}
+
+	// 准备告警事件信息
+	eventInfo, err := c.prepareEventInfo(event)
+	if err != nil {
+		return event, "", fmt.Errorf("failed to prepare event info: %v processor: %v", err, c)
+	}
+
+	// 调用AI模型生成总结
+	summary, err := c.generateAISummary(eventInfo)
+	if err != nil {
+		return event, "", fmt.Errorf("failed to generate AI summary: %v processor: %v", err, c)
+	}
+
+	// 将总结添加到annotations字段
+	if event.AnnotationsJSON == nil {
+		event.AnnotationsJSON = make(map[string]string)
+	}
+	event.AnnotationsJSON["ai_summary"] = summary
+
+	// 更新Annotations字段
+	b, err := json.Marshal(event.AnnotationsJSON)
+	if err != nil {
+		return event, "", fmt.Errorf("failed to marshal annotations: %v processor: %v", err, c)
+	}
+	event.Annotations = string(b)
+
+	return event, "", nil
+}
+
+func (c *AISummaryConfig) initHTTPClient() error {
+	transport := &http.Transport{
+		TLSClientConfig: &tls.Config{InsecureSkipVerify: c.SkipSSLVerify},
+	}
+
+	if c.Proxy != "" {
+		proxyURL, err := url.Parse(c.Proxy)
+		if err != nil {
+			return fmt.Errorf("failed to parse proxy url: %v", err)
+		}
+		transport.Proxy = http.ProxyURL(proxyURL)
+	}
+
+	c.Client = &http.Client{
+		Timeout:   time.Duration(c.Timeout) * time.Millisecond,
+		Transport: transport,
+	}
+	return nil
+}
+
+func (c *AISummaryConfig) prepareEventInfo(event *models.AlertCurEvent) (string, error) {
+	var defs = []string{
+		"{{$event := .}}",
+	}
+
+	text := strings.Join(append(defs, c.PromptTemplate), "")
+	t, err := template.New("prompt").Funcs(template.FuncMap(tplx.TemplateFuncMap)).Parse(text)
+	if err != nil {
+		return "", fmt.Errorf("failed to parse prompt template: %v", err)
+	}
+
+	var body bytes.Buffer
+	err = t.Execute(&body, event)
+	if err != nil {
+		return "", fmt.Errorf("failed to execute prompt template: %v", err)
+	}
+
+	return body.String(), nil
+}
+
+func (c *AISummaryConfig) generateAISummary(eventInfo string) (string, error) {
+	// 构建基础请求参数
+	reqParams := map[string]interface{}{
+		"model": c.ModelName,
+		"messages": []Message{
+			{
+				Role:    "user",
+				Content: eventInfo,
+			},
+		},
+	}
+
+	// 合并自定义参数
+	for k, v := range c.CustomParams {
+		reqParams[k] = v
+	}
+
+	// 序列化请求体
+	jsonData, err := json.Marshal(reqParams)
+	if err != nil {
+		return "", fmt.Errorf("failed to marshal request body: %v", err)
+	}
+
+	// 创建HTTP请求
+	req, err := http.NewRequest("POST", c.URL, bytes.NewBuffer(jsonData))
+	if err != nil {
+		return "", fmt.Errorf("failed to create request: %v", err)
+	}
+
+	// 设置请求头
+	req.Header.Set("Authorization", "Bearer "+c.APIKey)
+	req.Header.Set("Content-Type", "application/json")
+	for k, v := range c.Headers {
+		req.Header.Set(k, v)
+	}
+
+	// 发送请求
+	resp, err := c.Client.Do(req)
+	if err != nil {
+		return "", fmt.Errorf("failed to send request: %v", err)
+	}
+	defer resp.Body.Close()
+
+	// 检查响应状态码
+	if resp.StatusCode > HTTP_STATUS_SUCCESS_MAX {
+		body, _ := io.ReadAll(resp.Body)
+		return "", fmt.Errorf("unexpected status code: %d, body: %s", resp.StatusCode, string(body))
+	}
+
+	// 读取响应
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return "", fmt.Errorf("failed to read response body: %v", err)
+	}
+
+	// 解析响应
+	var chatResp ChatCompletionResponse
+	if err := json.Unmarshal(body, &chatResp); err != nil {
+		return "", fmt.Errorf("failed to unmarshal response: %v", err)
+	}
+
+	if len(chatResp.Choices) == 0 {
+		return "", fmt.Errorf("no response from AI model")
+	}
+
+	return chatResp.Choices[0].Message.Content, nil
+}

alert/pipeline/processor/aisummary/ai_summary_test.go

@@ -0,0 +1,69 @@
+package aisummary
+
+import (
+	"testing"
+
+	"github.com/ccfos/nightingale/v6/alert/pipeline/processor/callback"
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/ctx"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestAISummaryConfig_Process(t *testing.T) {
+	// 创建测试配置
+	config := &AISummaryConfig{
+		HTTPConfig: callback.HTTPConfig{
+			URL:           "https://generativelanguage.googleapis.com/v1beta/openai/chat/completions",
+			Timeout:       30000,
+			SkipSSLVerify: true,
+			Headers: map[string]string{
+				"Content-Type": "application/json",
+			},
+		},
+		ModelName:      "gemini-2.0-flash",
+		APIKey:         "*",
+		PromptTemplate: "告警规则：{{$event.RuleName}}\n严重程度：{{$event.Severity}}",
+		CustomParams: map[string]interface{}{
+			"temperature": 0.7,
+			"max_tokens":  2000,
+			"top_p":       0.9,
+		},
+	}
+
+	// 创建测试事件
+	event := &models.AlertCurEvent{
+		RuleName: "Test Rule",
+		Severity: 1,
+		TagsMap: map[string]string{
+			"host": "test-host",
+		},
+		AnnotationsJSON: map[string]string{
+			"description": "Test alert",
+		},
+	}
+
+	// 测试模板处理
+	eventInfo, err := config.prepareEventInfo(event)
+	assert.NoError(t, err)
+	assert.Contains(t, eventInfo, "Test Rule")
+	assert.Contains(t, eventInfo, "1")
+
+	// 测试配置初始化
+	processor, err := config.Init(config)
+	assert.NoError(t, err)
+	assert.NotNil(t, processor)
+
+	// 测试处理函数
+	result, _, err := processor.Process(&ctx.Context{}, event)
+	assert.NoError(t, err)
+	assert.NotNil(t, result)
+	assert.NotEmpty(t, result.AnnotationsJSON["ai_summary"])
+
+	// 展示处理结果
+	t.Log("\n=== 处理结果 ===")
+	t.Logf("告警规则: %s", result.RuleName)
+	t.Logf("严重程度: %d", result.Severity)
+	t.Logf("标签: %v", result.TagsMap)
+	t.Logf("原始注释: %v", result.AnnotationsJSON["description"])
+	t.Logf("AI总结: %s", result.AnnotationsJSON["ai_summary"])
+}

alert/pipeline/processor/callback/callback.go

@@ -0,0 +1,103 @@
+package callback
+
+import (
+	"crypto/tls"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"strings"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/alert/pipeline/processor/common"
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/ctx"
+	"github.com/toolkits/pkg/logger"
+)
+
+type HTTPConfig struct {
+	URL           string            `json:"url"`
+	Method        string            `json:"method,omitempty"`
+	Body          string            `json:"body,omitempty"`
+	Headers       map[string]string `json:"header"`
+	AuthUsername  string            `json:"auth_username"`
+	AuthPassword  string            `json:"auth_password"`
+	Timeout       int               `json:"timeout"` // 单位:ms
+	SkipSSLVerify bool              `json:"skip_ssl_verify"`
+	Proxy         string            `json:"proxy"`
+	Client        *http.Client      `json:"-"`
+}
+
+// RelabelConfig
+type CallbackConfig struct {
+	HTTPConfig
+}
+
+func init() {
+	models.RegisterProcessor("callback", &CallbackConfig{})
+}
+
+func (c *CallbackConfig) Init(settings interface{}) (models.Processor, error) {
+	result, err := common.InitProcessor[*CallbackConfig](settings)
+	return result, err
+}
+
+func (c *CallbackConfig) Process(ctx *ctx.Context, event *models.AlertCurEvent) (*models.AlertCurEvent, string, error) {
+	if c.Client == nil {
+		transport := &http.Transport{
+			TLSClientConfig: &tls.Config{InsecureSkipVerify: c.SkipSSLVerify},
+		}
+
+		if c.Proxy != "" {
+			proxyURL, err := url.Parse(c.Proxy)
+			if err != nil {
+				return event, "", fmt.Errorf("failed to parse proxy url: %v processor: %v", err, c)
+			} else {
+				transport.Proxy = http.ProxyURL(proxyURL)
+			}
+		}
+
+		c.Client = &http.Client{
+			Timeout:   time.Duration(c.Timeout) * time.Millisecond,
+			Transport: transport,
+		}
+	}
+
+	headers := make(map[string]string)
+	headers["Content-Type"] = "application/json"
+	for k, v := range c.Headers {
+		headers[k] = v
+	}
+
+	body, err := json.Marshal(event)
+	if err != nil {
+		return event, "", fmt.Errorf("failed to marshal event: %v processor: %v", err, c)
+	}
+
+	req, err := http.NewRequest("POST", c.URL, strings.NewReader(string(body)))
+	if err != nil {
+		return event, "", fmt.Errorf("failed to create request: %v processor: %v", err, c)
+	}
+
+	for k, v := range headers {
+		req.Header.Set(k, v)
+	}
+
+	if c.AuthUsername != "" && c.AuthPassword != "" {
+		req.SetBasicAuth(c.AuthUsername, c.AuthPassword)
+	}
+
+	resp, err := c.Client.Do(req)
+	if err != nil {
+		return event, "", fmt.Errorf("failed to send request: %v processor: %v", err, c)
+	}
+
+	b, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return event, "", fmt.Errorf("failed to read response body: %v processor: %v", err, c)
+	}
+
+	logger.Debugf("callback processor response body: %s", string(b))
+	return event, "callback success", nil
+}

alert/pipeline/processor/common/common.go

@@ -0,0 +1,24 @@
+package common
+
+import (
+	"encoding/json"
+)
+
+// InitProcessor 是一个通用的初始化处理器的方法
+// 使用泛型简化处理器初始化逻辑
+// T 必须是 models.Processor 接口的实现
+func InitProcessor[T any](settings interface{}) (T, error) {
+	var zero T
+	b, err := json.Marshal(settings)
+	if err != nil {
+		return zero, err
+	}
+
+	var result T
+	err = json.Unmarshal(b, &result)
+	if err != nil {
+		return zero, err
+	}
+
+	return result, nil
+}

alert/pipeline/processor/eventdrop/event_drop.go

@@ -0,0 +1,60 @@
+package eventdrop
+
+import (
+	"bytes"
+	"fmt"
+	"strings"
+	texttemplate "text/template"
+
+	"github.com/ccfos/nightingale/v6/alert/pipeline/processor/common"
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/ctx"
+	"github.com/ccfos/nightingale/v6/pkg/tplx"
+	"github.com/toolkits/pkg/logger"
+)
+
+type EventDropConfig struct {
+	Content string `json:"content"`
+}
+
+func init() {
+	models.RegisterProcessor("event_drop", &EventDropConfig{})
+}
+
+func (c *EventDropConfig) Init(settings interface{}) (models.Processor, error) {
+	result, err := common.InitProcessor[*EventDropConfig](settings)
+	return result, err
+}
+
+func (c *EventDropConfig) Process(ctx *ctx.Context, event *models.AlertCurEvent) (*models.AlertCurEvent, string, error) {
+	// 使用背景是可以根据此处理器，实现对事件进行更加灵活的过滤的逻辑
+	// 在标签过滤和属性过滤都不满足需求时可以使用
+	// 如果模板执行结果为 true，则删除该事件
+
+	var defs = []string{
+		"{{ $event := . }}",
+		"{{ $labels := .TagsMap }}",
+		"{{ $value := .TriggerValue }}",
+	}
+
+	text := strings.Join(append(defs, c.Content), "")
+
+	tpl, err := texttemplate.New("eventdrop").Funcs(tplx.TemplateFuncMap).Parse(text)
+	if err != nil {
+		return event, "", fmt.Errorf("processor failed to parse template: %v processor: %v", err, c)
+	}
+
+	var body bytes.Buffer
+	if err = tpl.Execute(&body, event); err != nil {
+		return event, "", fmt.Errorf("processor failed to execute template: %v processor: %v", err, c)
+	}
+
+	result := strings.TrimSpace(body.String())
+	logger.Infof("processor eventdrop result: %v", result)
+	if result == "true" {
+		logger.Infof("processor eventdrop drop event: %v", event)
+		return event, "drop event success", nil
+	}
+
+	return event, "drop event failed", nil
+}

alert/pipeline/processor/eventupdate/event_update.go

@@ -0,0 +1,96 @@
+package eventupdate
+
+import (
+	"crypto/tls"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"strings"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/alert/pipeline/processor/callback"
+	"github.com/ccfos/nightingale/v6/alert/pipeline/processor/common"
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/ctx"
+	"github.com/toolkits/pkg/logger"
+)
+
+// RelabelConfig
+type EventUpdateConfig struct {
+	callback.HTTPConfig
+}
+
+func init() {
+	models.RegisterProcessor("event_update", &EventUpdateConfig{})
+}
+
+func (c *EventUpdateConfig) Init(settings interface{}) (models.Processor, error) {
+	result, err := common.InitProcessor[*EventUpdateConfig](settings)
+	return result, err
+}
+
+func (c *EventUpdateConfig) Process(ctx *ctx.Context, event *models.AlertCurEvent) (*models.AlertCurEvent, string, error) {
+	if c.Client == nil {
+		transport := &http.Transport{
+			TLSClientConfig: &tls.Config{InsecureSkipVerify: c.SkipSSLVerify},
+		}
+
+		if c.Proxy != "" {
+			proxyURL, err := url.Parse(c.Proxy)
+			if err != nil {
+				return event, "", fmt.Errorf("failed to parse proxy url: %v processor: %v", err, c)
+			} else {
+				transport.Proxy = http.ProxyURL(proxyURL)
+			}
+		}
+
+		c.Client = &http.Client{
+			Timeout:   time.Duration(c.Timeout) * time.Millisecond,
+			Transport: transport,
+		}
+	}
+
+	headers := make(map[string]string)
+	headers["Content-Type"] = "application/json"
+	for k, v := range c.Headers {
+		headers[k] = v
+	}
+
+	body, err := json.Marshal(event)
+	if err != nil {
+		return event, "", fmt.Errorf("failed to marshal event: %v processor: %v", err, c)
+	}
+
+	req, err := http.NewRequest("POST", c.URL, strings.NewReader(string(body)))
+	if err != nil {
+		return event, "", fmt.Errorf("failed to create request: %v processor: %v", err, c)
+	}
+
+	for k, v := range headers {
+		req.Header.Set(k, v)
+	}
+
+	if c.AuthUsername != "" && c.AuthPassword != "" {
+		req.SetBasicAuth(c.AuthUsername, c.AuthPassword)
+	}
+
+	resp, err := c.Client.Do(req)
+	if err != nil {
+		return event, "", fmt.Errorf("failed to send request: %v processor: %v", err, c)
+	}
+
+	b, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, "", fmt.Errorf("failed to read response body: %v processor: %v", err, c)
+	}
+	logger.Debugf("event update processor response body: %s", string(b))
+
+	err = json.Unmarshal(b, &event)
+	if err != nil {
+		return event, "", fmt.Errorf("failed to unmarshal response body: %v processor: %v", err, c)
+	}
+
+	return event, "", nil
+}

alert/pipeline/processor/relabel/relabel.go

@@ -0,0 +1,107 @@
+package relabel
+
+import (
+	"fmt"
+	"regexp"
+	"strings"
+
+	"github.com/ccfos/nightingale/v6/alert/pipeline/processor/common"
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/ctx"
+	"github.com/ccfos/nightingale/v6/pushgw/pconf"
+	"github.com/ccfos/nightingale/v6/pushgw/writer"
+
+	"github.com/prometheus/common/model"
+	"github.com/prometheus/prometheus/prompb"
+)
+
+const (
+	REPLACE_DOT = "___"
+)
+
+// RelabelConfig
+type RelabelConfig struct {
+	SourceLabels  []string `json:"source_labels"`
+	Separator     string   `json:"separator"`
+	Regex         string   `json:"regex"`
+	RegexCompiled *regexp.Regexp
+	If            string `json:"if"`
+	IfRegex       *regexp.Regexp
+	Modulus       uint64 `json:"modulus"`
+	TargetLabel   string `json:"target_label"`
+	Replacement   string `json:"replacement"`
+	Action        string `json:"action"`
+}
+
+func init() {
+	models.RegisterProcessor("relabel", &RelabelConfig{})
+}
+
+func (r *RelabelConfig) Init(settings interface{}) (models.Processor, error) {
+	result, err := common.InitProcessor[*RelabelConfig](settings)
+	return result, err
+}
+
+func (r *RelabelConfig) Process(ctx *ctx.Context, event *models.AlertCurEvent) (*models.AlertCurEvent, string, error) {
+	sourceLabels := make([]model.LabelName, len(r.SourceLabels))
+	for i := range r.SourceLabels {
+		sourceLabels[i] = model.LabelName(strings.ReplaceAll(r.SourceLabels[i], ".", REPLACE_DOT))
+	}
+
+	relabelConfigs := []*pconf.RelabelConfig{
+		{
+			SourceLabels:  sourceLabels,
+			Separator:     r.Separator,
+			Regex:         r.Regex,
+			RegexCompiled: r.RegexCompiled,
+			If:            r.If,
+			IfRegex:       r.IfRegex,
+			Modulus:       r.Modulus,
+			TargetLabel:   r.TargetLabel,
+			Replacement:   r.Replacement,
+			Action:        r.Action,
+		},
+	}
+
+	EventRelabel(event, relabelConfigs)
+	return event, "", nil
+}
+
+func EventRelabel(event *models.AlertCurEvent, relabelConfigs []*pconf.RelabelConfig) {
+	labels := make([]prompb.Label, len(event.TagsJSON))
+	event.OriginalTagsJSON = make([]string, len(event.TagsJSON))
+	for i, tag := range event.TagsJSON {
+		label := strings.SplitN(tag, "=", 2)
+		if len(label) != 2 {
+			continue
+		}
+		event.OriginalTagsJSON[i] = tag
+
+		label[0] = strings.ReplaceAll(string(label[0]), ".", REPLACE_DOT)
+		labels[i] = prompb.Label{Name: label[0], Value: label[1]}
+	}
+
+	for i := 0; i < len(relabelConfigs); i++ {
+		if relabelConfigs[i].Replacement == "" {
+			relabelConfigs[i].Replacement = "$1"
+		}
+
+		if relabelConfigs[i].Separator == "" {
+			relabelConfigs[i].Separator = ";"
+		}
+
+		if relabelConfigs[i].Regex == "" {
+			relabelConfigs[i].Regex = "(.*)"
+		}
+	}
+
+	gotLabels := writer.Process(labels, relabelConfigs...)
+	event.TagsJSON = make([]string, len(gotLabels))
+	event.TagsMap = make(map[string]string, len(gotLabels))
+	for i, label := range gotLabels {
+		label.Name = strings.ReplaceAll(string(label.Name), REPLACE_DOT, ".")
+		event.TagsJSON[i] = fmt.Sprintf("%s=%s", label.Name, label.Value)
+		event.TagsMap[label.Name] = label.Value
+	}
+	event.Tags = strings.Join(event.TagsJSON, ",,")
+}

alert/process/process.go

@@ -14,14 +14,13 @@ import (
 	"github.com/ccfos/nightingale/v6/alert/common"
 	"github.com/ccfos/nightingale/v6/alert/dispatch"
 	"github.com/ccfos/nightingale/v6/alert/mute"
+	"github.com/ccfos/nightingale/v6/alert/pipeline/processor/relabel"
 	"github.com/ccfos/nightingale/v6/alert/queue"
 	"github.com/ccfos/nightingale/v6/memsto"
 	"github.com/ccfos/nightingale/v6/models"
 	"github.com/ccfos/nightingale/v6/pkg/ctx"
 	"github.com/ccfos/nightingale/v6/pkg/tplx"
-	"github.com/ccfos/nightingale/v6/pushgw/writer"
 
-	"github.com/prometheus/prometheus/prompb"
 	"github.com/robfig/cron/v3"
 	"github.com/toolkits/pkg/logger"
 	"github.com/toolkits/pkg/str"
@@ -61,11 +60,9 @@ type Processor struct {
 	pendingsUseByRecover *AlertCurEventMap
 	inhibit              bool
 
-	tagsMap    map[string]string
-	tagsArr    []string
-	target     string
-	targetNote string
-	groupName  string
+	tagsMap   map[string]string
+	tagsArr   []string
+	groupName string
 
 	alertRuleCache          *memsto.AlertRuleCacheType
 	TargetCache             *memsto.TargetCacheType
@@ -154,19 +151,30 @@ func (p *Processor) Handle(anomalyPoints []models.AnomalyPoint, from string, inh
 	eventsMap := make(map[string][]*models.AlertCurEvent)
 	for _, anomalyPoint := range anomalyPoints {
 		event := p.BuildEvent(anomalyPoint, from, now, ruleHash)
+		event.NotifyRuleIds = cachedRule.NotifyRuleIds
 		// 如果 event 被 mute 了,本质也是 fire 的状态,这里无论如何都添加到 alertingKeys 中,防止 fire 的事件自动恢复了
 		hash := event.Hash
 		alertingKeys[hash] = struct{}{}
-		isMuted, detail := mute.IsMuted(cachedRule, event, p.TargetCache, p.alertMuteCache)
+		isMuted, detail, muteId := mute.IsMuted(cachedRule, event, p.TargetCache, p.alertMuteCache)
 		if isMuted {
-			p.Stats.CounterMuteTotal.WithLabelValues(event.GroupName).Inc()
 			logger.Debugf("rule_eval:%s event:%v is muted, detail:%s", p.Key(), event, detail)
+			p.Stats.CounterMuteTotal.WithLabelValues(
+				fmt.Sprintf("%v", event.GroupName),
+				fmt.Sprintf("%v", p.rule.Id),
+				fmt.Sprintf("%v", muteId),
+				fmt.Sprintf("%v", p.datasourceId),
+			).Inc()
 			continue
 		}
 
 		if p.EventMuteHook(event) {
-			p.Stats.CounterMuteTotal.WithLabelValues(event.GroupName).Inc()
 			logger.Debugf("rule_eval:%s event:%v is muted by hook", p.Key(), event)
+			p.Stats.CounterMuteTotal.WithLabelValues(
+				fmt.Sprintf("%v", event.GroupName),
+				fmt.Sprintf("%v", p.rule.Id),
+				fmt.Sprintf("%v", 0),
+				fmt.Sprintf("%v", p.datasourceId),
+			).Inc()
 			continue
 		}
 
@@ -185,7 +193,7 @@ func (p *Processor) Handle(anomalyPoints []models.AnomalyPoint, from string, inh
 
 func (p *Processor) BuildEvent(anomalyPoint models.AnomalyPoint, from string, now int64, ruleHash string) *models.AlertCurEvent {
 	p.fillTags(anomalyPoint)
-	p.mayHandleIdent()
+
 	hash := Hash(p.rule.Id, p.datasourceId, anomalyPoint)
 	ds := p.datasourceCache.GetById(p.datasourceId)
 	var dsName string
@@ -205,8 +213,6 @@ func (p *Processor) BuildEvent(anomalyPoint models.AnomalyPoint, from string, no
 	event.DatasourceId = p.datasourceId
 	event.Cluster = dsName
 	event.Hash = hash
-	event.TargetIdent = p.target
-	event.TargetNote = p.targetNote
 	event.TriggerValue = anomalyPoint.ReadableValue()
 	event.TriggerValues = anomalyPoint.Values
 	event.TriggerValuesJson = models.EventTriggerValues{ValuesWithUnit: anomalyPoint.ValuesUnit}
@@ -238,15 +244,6 @@ func (p *Processor) BuildEvent(anomalyPoint models.AnomalyPoint, from string, no
 		logger.Warningf("unmarshal annotations json failed: %v, rule: %d", err, p.rule.Id)
 	}
 
-	if p.target != "" {
-		if pt, exist := p.TargetCache.Get(p.target); exist {
-			pt.GroupNames = p.BusiGroupCache.GetNamesByBusiGroupIds(pt.GroupIds)
-			event.Target = pt
-		} else {
-			logger.Infof("Target[ident: %s] doesn't exist in cache.", p.target)
-		}
-	}
-
 	if event.TriggerValues != "" && strings.Count(event.TriggerValues, "$") > 1 {
 		// TriggerValues 有多个变量，将多个变量都放到 TriggerValue 中
 		event.TriggerValue = event.TriggerValues
@@ -260,6 +257,19 @@ func (p *Processor) BuildEvent(anomalyPoint models.AnomalyPoint, from string, no
 
 	// 生成事件之后，立马进程 relabel 处理
 	Relabel(p.rule, event)
+
+	// 放到 Relabel(p.rule, event) 下面，为了处理 relabel 之后，标签里才出现 ident 的情况
+	p.mayHandleIdent(event)
+
+	if event.TargetIdent != "" {
+		if pt, exist := p.TargetCache.Get(event.TargetIdent); exist {
+			pt.GroupNames = p.BusiGroupCache.GetNamesByBusiGroupIds(pt.GroupIds)
+			event.Target = pt
+		} else {
+			logger.Infof("fill event target error, ident: %s doesn't exist in cache.", event.TargetIdent)
+		}
+	}
+
 	return event
 }
 
@@ -268,44 +278,15 @@ func Relabel(rule *models.AlertRule, event *models.AlertCurEvent) {
 		return
 	}
 
+	// need to keep the original label
+	event.OriginalTags = event.Tags
+	event.OriginalTagsJSON = event.TagsJSON
+
 	if len(rule.EventRelabelConfig) == 0 {
 		return
 	}
 
-	// need to keep the original label
-	event.OriginalTags = event.Tags
-	event.OriginalTagsJSON = make([]string, len(event.TagsJSON))
-
-	labels := make([]prompb.Label, len(event.TagsJSON))
-	for i, tag := range event.TagsJSON {
-		label := strings.SplitN(tag, "=", 2)
-		event.OriginalTagsJSON[i] = tag
-		labels[i] = prompb.Label{Name: label[0], Value: label[1]}
-	}
-
-	for i := 0; i < len(rule.EventRelabelConfig); i++ {
-		if rule.EventRelabelConfig[i].Replacement == "" {
-			rule.EventRelabelConfig[i].Replacement = "$1"
-		}
-
-		if rule.EventRelabelConfig[i].Separator == "" {
-			rule.EventRelabelConfig[i].Separator = ";"
-		}
-
-		if rule.EventRelabelConfig[i].Regex == "" {
-			rule.EventRelabelConfig[i].Regex = "(.*)"
-		}
-	}
-
-	// relabel process
-	relabels := writer.Process(labels, rule.EventRelabelConfig...)
-	event.TagsJSON = make([]string, len(relabels))
-	event.TagsMap = make(map[string]string, len(relabels))
-	for i, label := range relabels {
-		event.TagsJSON[i] = fmt.Sprintf("%s=%s", label.Name, label.Value)
-		event.TagsMap[label.Name] = label.Value
-	}
-	event.Tags = strings.Join(event.TagsJSON, ",,")
+	relabel.EventRelabel(event, rule.EventRelabelConfig)
 }
 
 func (p *Processor) HandleRecover(alertingKeys map[string]struct{}, now int64, inhibit bool) {
@@ -425,8 +406,8 @@ func (p *Processor) RecoverSingle(byRecover bool, hash string, now int64, value
 
 func (p *Processor) handleEvent(events []*models.AlertCurEvent) {
 	var fireEvents []*models.AlertCurEvent
-	// severity 初始为 4, 一定为遇到比自己优先级高的事件
-	severity := 4
+	// severity 初始为最低优先级, 一定为遇到比自己优先级高的事件
+	severity := models.SeverityLowest
 	for _, event := range events {
 		if event == nil {
 			continue
@@ -552,6 +533,12 @@ func (p *Processor) RecoverAlertCurEventFromDb() {
 	fireMap := make(map[string]*models.AlertCurEvent)
 	pendingsUseByRecoverMap := make(map[string]*models.AlertCurEvent)
 	for _, event := range curEvents {
+		alertRule := p.alertRuleCache.Get(event.RuleId)
+		if alertRule == nil {
+			continue
+		}
+		event.NotifyRuleIds = alertRule.NotifyRuleIds
+
 		if event.Cate == models.HOST {
 			target, exists := p.TargetCache.Get(event.TargetIdent)
 			if exists && target.EngineName != p.EngineName && !(p.ctx.IsCenter && target.EngineName == "") {
@@ -624,19 +611,19 @@ func (p *Processor) fillTags(anomalyPoint models.AnomalyPoint) {
 	p.tagsArr = labelMapToArr(tagsMap)
 }
 
-func (p *Processor) mayHandleIdent() {
+func (p *Processor) mayHandleIdent(event *models.AlertCurEvent) {
 	// handle ident
-	if ident, has := p.tagsMap["ident"]; has {
+	if ident, has := event.TagsMap["ident"]; has {
 		if target, exists := p.TargetCache.Get(ident); exists {
-			p.target = target.Ident
-			p.targetNote = target.Note
+			event.TargetIdent = target.Ident
+			event.TargetNote = target.Note
 		} else {
-			p.target = ident
-			p.targetNote = ""
+			event.TargetIdent = ident
+			event.TargetNote = ""
 		}
 	} else {
-		p.target = ""
-		p.targetNote = ""
+		event.TargetIdent = ""
+		event.TargetNote = ""
 	}
 }
 

alert/record/prom_rule.go

@@ -39,7 +39,7 @@ func NewRecordRuleContext(rule *models.RecordingRule, datasourceId int64, promCl
 		rule.CronPattern = fmt.Sprintf("@every %ds", rule.PromEvalInterval)
 	}
 
-	rrc.scheduler = cron.New(cron.WithSeconds())
+	rrc.scheduler = cron.New(cron.WithSeconds(), cron.WithChain(cron.SkipIfStillRunning(cron.DefaultLogger)))
 	_, err := rrc.scheduler.AddFunc(rule.CronPattern, func() {
 		rrc.Eval()
 	})
@@ -56,12 +56,13 @@ func (rrc *RecordRuleContext) Key() string {
 }
 
 func (rrc *RecordRuleContext) Hash() string {
-	return str.MD5(fmt.Sprintf("%d_%s_%s_%d_%s",
+	return str.MD5(fmt.Sprintf("%d_%s_%s_%d_%s_%s",
 		rrc.rule.Id,
 		rrc.rule.CronPattern,
 		rrc.rule.PromQl,
 		rrc.datasourceId,
 		rrc.rule.AppendTags,
+		rrc.rule.Name,
 	))
 }
 

alert/router/router_event.go

@@ -40,8 +40,8 @@ func (rt *Router) pushEventToQueue(c *gin.Context) {
 
 		event.TagsMap[arr[0]] = arr[1]
 	}
-
-	if mute.EventMuteStrategy(event, rt.AlertMuteCache) {
+	hit, _ :=  mute.EventMuteStrategy(event, rt.AlertMuteCache)
+	if hit {
 		logger.Infof("event_muted: rule_id=%d %s", event.RuleId, event.Hash)
 		ginx.NewRender(c).Message(nil)
 		return

alert/sender/callback.go

@@ -135,14 +135,14 @@ func (c *DefaultCallBacker) CallBack(ctx CallBackContext) {
 func doSendAndRecord(ctx *ctx.Context, url, token string, body interface{}, channel string,
 	stats *astats.Stats, events []*models.AlertCurEvent) {
 	res, err := doSend(url, body, channel, stats)
-	NotifyRecord(ctx, events, channel, token, res, err)
+	NotifyRecord(ctx, events, 0, channel, token, res, err)
 }
 
-func NotifyRecord(ctx *ctx.Context, evts []*models.AlertCurEvent, channel, target, res string, err error) {
+func NotifyRecord(ctx *ctx.Context, evts []*models.AlertCurEvent, notifyRuleID int64, channel, target, res string, err error) {
 	// 一个通知可能对应多个 event，都需要记录
 	notis := make([]*models.NotificaitonRecord, 0, len(evts))
 	for _, evt := range evts {
-		noti := models.NewNotificationRecord(evt, channel, target)
+		noti := models.NewNotificationRecord(evt, notifyRuleID, channel, target)
 		if err != nil {
 			noti.SetStatus(models.NotiStatusFailure)
 			noti.SetDetails(err.Error())

alert/sender/email.go

@@ -205,7 +205,7 @@ func startEmailSender(ctx *ctx.Context, smtp aconf.SMTPConfig) {
 				if err == nil {
 					msg = "ok"
 				}
-				NotifyRecord(ctx, m.events, models.Email, to, msg, err)
+				NotifyRecord(ctx, m.events, 0, models.Email, to, msg, err)
 			}
 
 			size++

alert/sender/ibex.go

@@ -30,12 +30,14 @@ type IbexCallBacker struct {
 
 func (c *IbexCallBacker) CallBack(ctx CallBackContext) {
 	if len(ctx.CallBackURL) == 0 || len(ctx.Events) == 0 {
+		logger.Warningf("event_callback_ibex: url or events is empty, url: %s, events: %+v", ctx.CallBackURL, ctx.Events)
 		return
 	}
 
 	event := ctx.Events[0]
 
 	if event.IsRecovered {
+		logger.Infof("event_callback_ibex: event is recovered, event: %+v", event)
 		return
 	}
 
@@ -43,8 +45,9 @@ func (c *IbexCallBacker) CallBack(ctx CallBackContext) {
 }
 
 func (c *IbexCallBacker) handleIbex(ctx *ctx.Context, url string, event *models.AlertCurEvent) {
+	logger.Infof("event_callback_ibex: url: %s, event: %+v", url, event)
 	if imodels.DB() == nil && ctx.IsCenter {
-		logger.Warning("event_callback_ibex: db is nil")
+		logger.Warningf("event_callback_ibex: db is nil, event: %+v", event)
 		return
 	}
 
@@ -63,17 +66,23 @@ func (c *IbexCallBacker) handleIbex(ctx *ctx.Context, url string, event *models.
 
 	id, err := strconv.ParseInt(idstr, 10, 64)
 	if err != nil {
-		logger.Errorf("event_callback_ibex: failed to parse url: %s", url)
+		logger.Errorf("event_callback_ibex: failed to parse url: %s event: %+v", url, event)
 		return
 	}
 
 	if host == "" {
 		// 用户在callback url中没有传入host，就从event中解析
 		host = event.TargetIdent
+
+		if host == "" {
+			if ident, has := event.TagsMap["ident"]; has {
+				host = ident
+			}
+		}
 	}
 
 	if host == "" {
-		logger.Error("event_callback_ibex: failed to get host")
+		logger.Errorf("event_callback_ibex: failed to get host, id: %d, event: %+v", id, event)
 		return
 	}
 
@@ -83,21 +92,23 @@ func (c *IbexCallBacker) handleIbex(ctx *ctx.Context, url string, event *models.
 func CallIbex(ctx *ctx.Context, id int64, host string,
 	taskTplCache *memsto.TaskTplCache, targetCache *memsto.TargetCacheType,
 	userCache *memsto.UserCacheType, event *models.AlertCurEvent) {
+	logger.Infof("event_callback_ibex: id: %d, host: %s, event: %+v", id, host, event)
+
 	tpl := taskTplCache.Get(id)
 	if tpl == nil {
-		logger.Errorf("event_callback_ibex: no such tpl(%d)", id)
+		logger.Errorf("event_callback_ibex: no such tpl(%d), event: %+v", id, event)
 		return
 	}
 	// check perm
 	// tpl.GroupId - host - account 三元组校验权限
 	can, err := canDoIbex(tpl.UpdateBy, tpl, host, targetCache, userCache)
 	if err != nil {
-		logger.Errorf("event_callback_ibex: check perm fail: %v", err)
+		logger.Errorf("event_callback_ibex: check perm fail: %v, event: %+v", err, event)
 		return
 	}
 
 	if !can {
-		logger.Errorf("event_callback_ibex: user(%s) no permission", tpl.UpdateBy)
+		logger.Errorf("event_callback_ibex: user(%s) no permission, event: %+v", tpl.UpdateBy, event)
 		return
 	}
 
@@ -122,7 +133,7 @@ func CallIbex(ctx *ctx.Context, id int64, host string,
 
 	tags, err := json.Marshal(tagsMap)
 	if err != nil {
-		logger.Errorf("event_callback_ibex: failed to marshal tags to json: %v", tagsMap)
+		logger.Errorf("event_callback_ibex: failed to marshal tags to json: %v, event: %+v", tagsMap, event)
 		return
 	}
 
@@ -145,7 +156,7 @@ func CallIbex(ctx *ctx.Context, id int64, host string,
 
 	id, err = TaskAdd(in, tpl.UpdateBy, ctx.IsCenter)
 	if err != nil {
-		logger.Errorf("event_callback_ibex: call ibex fail: %v", err)
+		logger.Errorf("event_callback_ibex: call ibex fail: %v, event: %+v", err, event)
 		return
 	}
 
@@ -167,7 +178,7 @@ func CallIbex(ctx *ctx.Context, id int64, host string,
 	}
 
 	if err = record.Add(ctx); err != nil {
-		logger.Errorf("event_callback_ibex: persist task_record fail: %v", err)
+		logger.Errorf("event_callback_ibex: persist task_record fail: %v, event: %+v", err, event)
 	}
 }
 
@@ -187,7 +198,7 @@ func canDoIbex(username string, tpl *models.TaskTpl, host string, targetCache *m
 
 func TaskAdd(f models.TaskForm, authUser string, isCenter bool) (int64, error) {
 	if storage.Cache == nil {
-		logger.Warning("event_callback_ibex: redis cache is nil")
+		logger.Warningf("event_callback_ibex: redis cache is nil, task: %+v", f)
 		return 0, fmt.Errorf("redis cache is nil")
 	}
 

alert/sender/mm.go

@@ -74,7 +74,7 @@ func SendMM(ctx *ctx.Context, message MatterMostMessage, events []*models.AlertC
 		u, err := url.Parse(message.Tokens[i])
 		if err != nil {
 			logger.Errorf("mm_sender: failed to parse error=%v", err)
-			NotifyRecord(ctx, events, channel, message.Tokens[i], "", err)
+			NotifyRecord(ctx, events, 0, channel, message.Tokens[i], "", err)
 			continue
 		}
 

alert/sender/plugin.go

@@ -104,7 +104,7 @@ func alertingCallScript(ctx *ctx.Context, stdinBytes []byte, notifyScript models
 		res = res[:validLen] + "..."
 	}
 
-	NotifyRecord(ctx, []*models.AlertCurEvent{event}, channel, cmd.String(), res, buildErr(err, isTimeout))
+	NotifyRecord(ctx, []*models.AlertCurEvent{event}, 0, channel, cmd.String(), res, buildErr(err, isTimeout))
 
 	if isTimeout {
 		if err == nil {

alert/sender/telegram.go

@@ -72,7 +72,7 @@ func SendTelegram(ctx *ctx.Context, message TelegramMessage, events []*models.Al
 	for i := 0; i < len(message.Tokens); i++ {
 		if !strings.Contains(message.Tokens[i], "/") && !strings.HasPrefix(message.Tokens[i], "https://") {
 			logger.Errorf("telegram_sender: result=fail invalid token=%s", message.Tokens[i])
-			NotifyRecord(ctx, events, channel, message.Tokens[i], "", errors.New("invalid token"))
+			NotifyRecord(ctx, events, 0, channel, message.Tokens[i], "", errors.New("invalid token"))
 			continue
 		}
 		var url string

alert/sender/webhook.go

@@ -100,7 +100,7 @@ func SingleSendWebhooks(ctx *ctx.Context, webhooks map[string]*models.Webhook, e
 		retryCount := 0
 		for retryCount < 3 {
 			needRetry, res, err := sendWebhook(conf, event, stats)
-			NotifyRecord(ctx, []*models.AlertCurEvent{event}, "webhook", conf.Url, res, err)
+			NotifyRecord(ctx, []*models.AlertCurEvent{event}, 0, "webhook", conf.Url, res, err)
 			if !needRetry {
 				break
 			}
@@ -170,7 +170,7 @@ func StartConsumer(ctx *ctx.Context, queue *WebhookQueue, popSize int, webhook *
 			retryCount := 0
 			for retryCount < webhook.RetryCount {
 				needRetry, res, err := sendWebhook(webhook, events, stats)
-				go NotifyRecord(ctx, events, "webhook", webhook.Url, res, err)
+				go NotifyRecord(ctx, events, 0, "webhook", webhook.Url, res, err)
 				if !needRetry {
 					break
 				}

center/cconf/ops.go

@@ -85,212 +85,221 @@ func MergeOperationConf() error {
 const (
 	builtInOps = `
 ops:
-- name: dashboards
-  cname: Dashboards
-  ops:
-    - name: "/dashboards"
-      cname: View Dashboards
-    - name: "/dashboards/add"
-      cname: Add Dashboard
-    - name: "/dashboards/put"
-      cname: Modify Dashboard
-    - name: "/dashboards/del"
-      cname: Delete Dashboard
-    - name: "/embedded-dashboards/put"
-      cname: Modify Embedded Dashboard
-    - name: "/embedded-dashboards"
-      cname: View Embedded Dashboard
-    - name: "/public-dashboards"
-      cname: View Public Dashboard
-
-- name: metric
-  cname: Time Series Metrics
-  ops:
-    - name: "/metric/explorer"
-      cname: View Metric Data
-    - name: "/object/explorer"
-      cname: View Object Data
-
-- name: builtin-metrics
-  cname: Metric Views
-  ops:
-    - name: "/metrics-built-in"
-      cname: View Built-in Metrics
-    - name: "/builtin-metrics/add"
-      cname: Add Built-in Metric
-    - name: "/builtin-metrics/put"
-      cname: Modify Built-in Metric
-    - name: "/builtin-metrics/del"
-      cname: Delete Built-in Metric
-
-- name: recording-rules
-  cname: Recording Rule Management
-  ops:
-    - name: "/recording-rules"
-      cname: View Recording Rules
-    - name: "/recording-rules/add"
-      cname: Add Recording Rule
-    - name: "/recording-rules/put"
-      cname: Modify Recording Rule
-    - name: "/recording-rules/del"
-      cname: Delete Recording Rule
-
-- name: log
-  cname: Log Analysis
-  ops:
-    - name: "/log/explorer"
-      cname: View Logs
-    - name: "/log/index-patterns"
-      cname: View Index Patterns
-
-- name: alert
-  cname: Alert Rules
-  ops:
-    - name: "/alert-rules"
-      cname: View Alert Rules
-    - name: "/alert-rules/add"
-      cname: Add Alert Rule
-    - name: "/alert-rules/put"
-      cname: Modify Alert Rule
-    - name: "/alert-rules/del"
-      cname: Delete Alert Rule
-
-- name: alert-mutes
-  cname: Alert Silence Management
-  ops:
-    - name: "/alert-mutes"
-      cname: View Alert Silences
-    - name: "/alert-mutes/add"
-      cname: Add Alert Silence
-    - name: "/alert-mutes/put"
-      cname: Modify Alert Silence
-    - name: "/alert-mutes/del"
-      cname: Delete Alert Silence
-  
-- name: alert-subscribes
-  cname: Alert Subscription Management
-  ops:
-    - name: "/alert-subscribes"
-      cname: View Alert Subscriptions
-    - name: "/alert-subscribes/add"
-      cname: Add Alert Subscription
-    - name: "/alert-subscribes/put"
-      cname: Modify Alert Subscription
-    - name: "/alert-subscribes/del"
-      cname: Delete Alert Subscription
-
-- name: alert-events  
-  cname: Alert Event Management
-  ops:
-    - name: "/alert-cur-events"
-      cname: View Current Alerts
-    - name: "/alert-cur-events/del"
-      cname: Delete Current Alert
-    - name: "/alert-his-events"
-      cname: View Historical Alerts
-
-- name: notification
-  cname: Alert Notification
-  ops:
-    - name: "/help/notification-settings"
-      cname: View Notification Settings
-    - name: "/help/notification-tpls"
-      cname: View Notification Templates
-
-- name: job
-  cname: Task Management
-  ops:
-    - name: "/job-tpls"
-      cname: View Task Templates
-    - name: "/job-tpls/add"
-      cname: Add Task Template
-    - name: "/job-tpls/put"
-      cname: Modify Task Template
-    - name: "/job-tpls/del"
-      cname: Delete Task Template
-    - name: "/job-tasks"
-      cname: View Task Instances
-    - name: "/job-tasks/add"
-      cname: Add Task Instance
-    - name: "/job-tasks/put"
-      cname: Modify Task Instance
-
-- name: targets
+- name: Infrastructure
   cname: Infrastructure
   ops:
-    - name: "/targets"
-      cname: View Objects
-    - name: "/targets/add"
-      cname: Add Object
-    - name: "/targets/put"
-      cname: Modify Object
-    - name: "/targets/del"
-      cname: Delete Object
-    - name: "/targets/bind"
-      cname: Bind Object
+    - name: /targets
+      cname: Host - View
+    - name: /targets/put
+      cname: Host - Modify
+    - name: /targets/del
+      cname: Host - Delete
+    - name: /targets/bind
+      cname: Host - Bind Uncategorized
 
-- name: user
-  cname: User Management
+- name: Explorer
+  cname: Explorer
   ops:
-    - name: "/users"
-      cname: View User List
-    - name: "/user-groups"
-      cname: View User Groups
-    - name: "/user-groups/add"
-      cname: Add User Group
-    - name: "/user-groups/put"
-      cname: Modify User Group
-    - name: "/user-groups/del"
-      cname: Delete User Group
+    - name: /metric/explorer
+      cname: Metrics Explorer
+    - name: /object/explorer
+      cname: Quick View
+    - name: /metrics-built-in
+      cname: Built-in Metric - View
+    - name: /builtin-metrics/add
+      cname: Built-in Metric - Add
+    - name: /builtin-metrics/put
+      cname: Built-in Metric - Modify
+    - name: /builtin-metrics/del
+      cname: Built-in Metric - Delete
+    - name: /recording-rules
+      cname: Recording Rule - View
+    - name: /recording-rules/add
+      cname: Recording Rule - Add
+    - name: /recording-rules/put
+      cname: Recording Rule - Modify
+    - name: /recording-rules/del
+      cname: Recording Rule - Delete
+    - name: /log/explorer
+      cname: Logs Explorer
+    - name: /log/index-patterns # 前端有个管理索引模式的页面，所以需要一个权限点来控制，后面应该改成侧拉板
+      cname: Index Pattern - View
+    - name: /log/index-patterns/add
+      cname: Index Pattern - Add
+    - name: /log/index-patterns/put
+      cname: Index Pattern - Modify
+    - name: /log/index-patterns/del
+      cname: Index Pattern - Delete
+    - name: /dashboards
+      cname: Dashboard - View
+    - name: /dashboards/add
+      cname: Dashboard - Add
+    - name: /dashboards/put
+      cname: Dashboard - Modify
+    - name: /dashboards/del
+      cname: Dashboard - Delete
+    - name: /public-dashboards
+      cname: Dashboard - View Public
 
-- name: busi-groups
-  cname: Business Group Management
+- name: alerting
+  cname: Alerting
   ops:
-    - name: "/busi-groups"
-      cname: View Business Groups
-    - name: "/busi-groups/add"
-      cname: Add Business Group
-    - name: "/busi-groups/put"
-      cname: Modify Business Group
-    - name: "/busi-groups/del"
-      cname: Delete Business Group
+    - name: /alert-rules
+      cname: Alerting Rule - View
+    - name: /alert-rules/add
+      cname: Alerting Rule - Add
+    - name: /alert-rules/put
+      cname: Alerting Rule - Modify
+    - name: /alert-rules/del
+      cname: Alerting Rule - Delete
+    - name: /alert-mutes
+      cname: Mutting Rule - View
+    - name: /alert-mutes/add
+      cname: Mutting Rule - Add
+    - name: /alert-mutes/put
+      cname: Mutting Rule - Modify
+    - name: /alert-mutes/del
+      cname: Mutting Rule - Delete
+    - name: /alert-subscribes
+      cname: Subscribing Rule - View
+    - name: /alert-subscribes/add
+      cname: Subscribing Rule - Add
+    - name: /alert-subscribes/put
+      cname: Subscribing Rule - Modify
+    - name: /alert-subscribes/del
+      cname: Subscribing Rule - Delete
+    - name: /job-tpls
+      cname: Self-healing-Script - View
+    - name: /job-tpls/add
+      cname: Self-healing-Script - Add
+    - name: /job-tpls/put
+      cname: Self-healing-Script - Modify
+    - name: /job-tpls/del
+      cname: Self-healing-Script - Delete
+    - name: /job-tasks
+      cname: Self-healing-Job - View
+    - name: /job-tasks/add
+      cname: Self-healing-Job - Add
+    - name: /job-tasks/put
+      cname: Self-healing-Job - Modify
+    - name: /alert-cur-events
+      cname: Active Event - View
+    - name: /alert-cur-events/del
+      cname: Active Event - Delete
+    - name: /alert-his-events
+      cname: Historical Event - View
 
-- name: permissions
-  cname: Permission Management
+- name: Notification
+  cname: Notification
   ops:
-    - name: "/permissions"
-      cname: View Permission Settings
+    - name: /notification-rules
+      cname: Notification Rule - View
+    - name: /notification-rules/add
+      cname: Notification Rule - Add
+    - name: /notification-rules/put
+      cname: Notification Rule - Modify
+    - name: /notification-rules/del
+      cname: Notification Rule - Delete
+    - name: /notification-channels
+      cname: Media Type - View
+    - name: /notification-channels/add
+      cname: Media Type - Add
+    - name: /notification-channels/put
+      cname: Media Type - Modify
+    - name: /notification-channels/del
+      cname: Media Type - Delete
+    - name: /notification-templates
+      cname: Message Template - View
+    - name: /notification-templates/add
+      cname: Message Template - Add
+    - name: /notification-templates/put
+      cname: Message Template - Modify
+    - name: /notification-templates/del
+      cname: Message Template - Delete
+    - name: /event-pipelines
+      cname: Event Pipeline - View
+    - name: /event-pipelines/add
+      cname: Event Pipeline - Add
+    - name: /event-pipelines/put
+      cname: Event Pipeline - Modify
+    - name: /event-pipelines/del
+      cname: Event Pipeline - Delete
+    - name: /help/notification-settings # 用于控制老版本的通知设置菜单是否展示
+      cname: Notification Settings - View
+    - name: /help/notification-tpls # 用于控制老版本的通知模板菜单是否展示
+      cname: Notification Templates - View
 
-- name: built-in-components
-  cname: Template Center
+- name: Integrations
+  cname: Integrations
   ops:
-    - name: "/built-in-components"
-      cname: View Built-in Components
-    - name: "/built-in-components/add"
-      cname: Add Built-in Component
-    - name: "/built-in-components/put"
-      cname: Modify Built-in Component
-    - name: "/built-in-components/del"
-      cname: Delete Built-in Component
+    - name: /datasources # 用于控制能否看到数据源列表页面的菜单。只有 Admin 才能修改、删除数据源
+      cname: Data Source - View
+    - name: /components
+      cname: Component - View
+    - name: /components/add
+      cname: Component - Add
+    - name: /components/put
+      cname: Component - Modify
+    - name: /components/del
+      cname: Component - Delete
+    - name: /embedded-products
+      cname: Embedded Product - View
+    - name: /embedded-product/add
+      cname: Embedded Product - Add
+    - name: /embedded-product/put
+      cname: Embedded Product - Modify
+    - name: /embedded-product/delete
+      cname: Embedded Product - Delete
 
-- name: datasource
-  cname: Data Source Management
+- name: Organization
+  cname: Organization
   ops:
-    - name: "/help/source"
-      cname: View Data Source Configuration
+    - name: /users
+      cname: User - View
+    - name: /users/add
+      cname: User - Add
+    - name: /users/put
+      cname: User - Modify
+    - name: /users/del
+      cname: User - Delete
+    - name: /user-groups
+      cname: Team - View
+    - name: /user-groups/add
+      cname: Team - Add
+    - name: /user-groups/put
+      cname: Team - Modify
+    - name: /user-groups/del
+      cname: Team - Delete
+    - name: /busi-groups
+      cname: Business Group - View
+    - name: /busi-groups/add
+      cname: Business Group - Add
+    - name: /busi-groups/put
+      cname: Business Group - Modify
+    - name: /busi-groups/del
+      cname: Business Group - Delete
+    - name: /roles
+      cname: Role - View
+    - name: /roles/add
+      cname: Role - Add
+    - name: /roles/put
+      cname: Role - Modify
+    - name: /roles/del
+      cname: Role - Delete
 
-- name: system
-  cname: System Information
+- name: System Settings
+  cname: System Settings
   ops:
-    - name: "/help/variable-configs"
-      cname: View Variable Configuration
-    - name: "/help/version"
-      cname: View Version Information
-    - name: "/help/servers"
-      cname: View Server Information
-    - name: "/help/sso"
-      cname: View SSO Configuration
-    - name: "/site-settings"
+    - name: /system/site-settings # 仅用于控制能否展示菜单，只有 Admin 才能修改、删除
       cname: View Site Settings
+    - name: /system/variable-settings
+      cname: View Variable Settings
+    - name: /system/sso-settings
+      cname: View SSO Settings
+    - name: /system/alerting-engines
+      cname: View Alerting Engines
+    - name: /system/version
+      cname: View Product Version
+
 `
 )

center/cconf/plugin.go

@@ -25,4 +25,22 @@ var Plugins = []Plugin{
 		Type:     "tdengine",
 		TypeName: "TDengine",
 	},
+	{
+		Id:       5,
+		Category: "logging",
+		Type:     "ck",
+		TypeName: "ClickHouse",
+	},
+	{
+		Id:       6,
+		Category: "timeseries",
+		Type:     "mysql",
+		TypeName: "MySQL",
+	},
+	{
+		Id:       7,
+		Category: "timeseries",
+		Type:     "pgsql",
+		TypeName: "PostgreSQL",
+	},
 }

center/center.go

@@ -13,7 +13,6 @@ import (
 	alertrt "github.com/ccfos/nightingale/v6/alert/router"
 	"github.com/ccfos/nightingale/v6/center/cconf"
 	"github.com/ccfos/nightingale/v6/center/cconf/rsa"
-	"github.com/ccfos/nightingale/v6/center/cstats"
 	"github.com/ccfos/nightingale/v6/center/integration"
 	"github.com/ccfos/nightingale/v6/center/metas"
 	centerrt "github.com/ccfos/nightingale/v6/center/router"
@@ -60,7 +59,6 @@ func Initialize(configDir string, cryptoKey string) (func(), error) {
 	}
 
 	i18nx.Init(configDir)
-	cstats.Init()
 	flashduty.Init(config.Center.FlashDuty)
 
 	db, err := storage.New(config.DB)
@@ -86,11 +84,18 @@ func Initialize(configDir string, cryptoKey string) (func(), error) {
 	}
 
 	metas := metas.New(redis)
-	idents := idents.New(ctx, redis)
+	idents := idents.New(ctx, redis, config.Pushgw)
 
 	syncStats := memsto.NewSyncStats()
 	alertStats := astats.NewSyncStats()
 
+	if config.Center.MigrateBusiGroupLabel || models.CanMigrateBg(ctx) {
+		models.MigrateBg(ctx, config.Pushgw.BusiGroupLabelKey)
+	}
+	if models.CanMigrateEP(ctx) {
+		models.MigrateEP(ctx)
+	}
+
 	configCache := memsto.NewConfigCache(ctx, syncStats, config.HTTP.RSA.RSAPrivateKey, config.HTTP.RSA.RSAPassWord)
 	busiGroupCache := memsto.NewBusiGroupCache(ctx, syncStats)
 	targetCache := memsto.NewTargetCache(ctx, syncStats, redis)
@@ -102,6 +107,9 @@ func Initialize(configDir string, cryptoKey string) (func(), error) {
 	userGroupCache := memsto.NewUserGroupCache(ctx, syncStats)
 	taskTplCache := memsto.NewTaskTplCache(ctx)
 	configCvalCache := memsto.NewCvalCache(ctx, syncStats)
+	notifyRuleCache := memsto.NewNotifyRuleCache(ctx, syncStats)
+	notifyChannelCache := memsto.NewNotifyChannelCache(ctx, syncStats)
+	messageTemplateCache := memsto.NewMessageTemplateCache(ctx, syncStats)
 	userTokenCache := memsto.NewUserTokenCache(ctx, syncStats)
 
 	sso := sso.Init(config.Center, ctx, configCache)
@@ -113,7 +121,7 @@ func Initialize(configDir string, cryptoKey string) (func(), error) {
 
 	macros.RegisterMacro(macros.MacroInVain)
 	dscache.Init(ctx, false)
-	alert.Start(config.Alert, config.Pushgw, syncStats, alertStats, externalProcessors, targetCache, busiGroupCache, alertMuteCache, alertRuleCache, notifyConfigCache, taskTplCache, dsCache, ctx, promClients, userCache, userGroupCache)
+	alert.Start(config.Alert, config.Pushgw, syncStats, alertStats, externalProcessors, targetCache, busiGroupCache, alertMuteCache, alertRuleCache, notifyConfigCache, taskTplCache, dsCache, ctx, promClients, userCache, userGroupCache, notifyRuleCache, notifyChannelCache, messageTemplateCache)
 
 	writers := writer.NewWriters(config.Pushgw)
 
@@ -127,12 +135,6 @@ func Initialize(configDir string, cryptoKey string) (func(), error) {
 		redis, sso, ctx, metas, idents, targetCache, userCache, userGroupCache, userTokenCache)
 	pushgwRouter := pushgwrt.New(config.HTTP, config.Pushgw, config.Alert, targetCache, busiGroupCache, idents, metas, writers, ctx)
 
-	go func() {
-		if config.Center.MigrateBusiGroupLabel || models.CanMigrateBg(ctx) {
-			models.MigrateBg(ctx, pushgwRouter.Pushgw.BusiGroupLabelKey)
-		}
-	}()
-
 	r := httpx.GinEngine(config.Global.RunMode, config.HTTP, configCvalCache.PrintBodyPaths, configCvalCache.PrintAccessLog)
 
 	centerRouter.Config(r)

center/cstats/stats.go

@@ -6,40 +6,49 @@ import (
 	"github.com/prometheus/client_golang/prometheus"
 )
 
-const Service = "n9e-center"
+const (
+	namespace = "n9e"
+	subsystem = "center"
+)
 
 var (
-	labels = []string{"service", "code", "path", "method"}
-
-	uptime = prometheus.NewCounterVec(
+	uptime = prometheus.NewCounter(
 		prometheus.CounterOpts{
-			Name: "uptime",
-			Help: "HTTP service uptime.",
-		}, []string{"service"},
-	)
-
-	RequestCounter = prometheus.NewCounterVec(
-		prometheus.CounterOpts{
-			Name: "http_request_count_total",
-			Help: "Total number of HTTP requests made.",
-		}, labels,
+			Namespace: namespace,
+			Subsystem: subsystem,
+			Name:      "uptime",
+			Help:      "HTTP service uptime.",
+		},
 	)
 
 	RequestDuration = prometheus.NewHistogramVec(
 		prometheus.HistogramOpts{
-			Buckets: []float64{.01, .1, 1, 10},
-			Name:    "http_request_duration_seconds",
-			Help:    "HTTP request latencies in seconds.",
-		}, labels,
+			Namespace: namespace,
+			Subsystem: subsystem,
+			Buckets:   prometheus.DefBuckets,
+			Name:      "http_request_duration_seconds",
+			Help:      "HTTP request latencies in seconds.",
+		}, []string{"code", "path", "method"},
+	)
+
+	RedisOperationLatency = prometheus.NewHistogramVec(
+		prometheus.HistogramOpts{
+			Namespace: namespace,
+			Subsystem: subsystem,
+			Name:      "redis_operation_latency_seconds",
+			Help:      "Histogram of latencies for Redis operations",
+			Buckets:   []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5},
+		},
+		[]string{"operation", "status"},
 	)
 )
 
-func Init() {
+func init() {
 	// Register the summary and the histogram with Prometheus's default registry.
 	prometheus.MustRegister(
 		uptime,
-		RequestCounter,
 		RequestDuration,
+		RedisOperationLatency,
 	)
 
 	go recordUptime()
@@ -48,6 +57,6 @@ func Init() {
 // recordUptime increases service uptime per second.
 func recordUptime() {
 	for range time.Tick(time.Second) {
-		uptime.WithLabelValues(Service).Inc()
+		uptime.Inc()
 	}
 }

center/integration/init.go

@@ -22,6 +22,14 @@ func Init(ctx *ctx.Context, builtinIntegrationsDir string) {
 		return
 	}
 
+	if res, err := models.ConfigsSelectByCkey(ctx, "disable_integration_init"); err != nil {
+		logger.Error("fail to get value 'disable_integration_init' from configs", err)
+		return
+	} else if len(res) != 0 {
+		logger.Info("disable_integration_init is set, skip integration init")
+		return
+	}
+
 	fp := builtinIntegrationsDir
 	if fp == "" {
 		fp = path.Join(runner.Cwd, "integrations")

center/metas/metas.go

@@ -6,6 +6,7 @@ import (
 	"sync"
 	"time"
 
+	"github.com/ccfos/nightingale/v6/center/cstats"
 	"github.com/ccfos/nightingale/v6/models"
 	"github.com/ccfos/nightingale/v6/storage"
 
@@ -115,15 +116,23 @@ func (s *Set) updateTargets(m map[string]models.HostMeta) error {
 		}
 		newMap[models.WrapIdent(ident)] = meta
 	}
+
+	start := time.Now()
 	err := storage.MSet(context.Background(), s.redis, newMap)
 	if err != nil {
+		cstats.RedisOperationLatency.WithLabelValues("mset_target_meta", "fail").Observe(time.Since(start).Seconds())
 		return err
+	} else {
+		cstats.RedisOperationLatency.WithLabelValues("mset_target_meta", "success").Observe(time.Since(start).Seconds())
 	}
 
 	if len(extendMap) > 0 {
 		err = storage.MSet(context.Background(), s.redis, extendMap)
 		if err != nil {
+			cstats.RedisOperationLatency.WithLabelValues("mset_target_extend", "fail").Observe(time.Since(start).Seconds())
 			return err
+		} else {
+			cstats.RedisOperationLatency.WithLabelValues("mset_target_extend", "success").Observe(time.Since(start).Seconds())
 		}
 	}
 

center/router/router.go

@@ -93,10 +93,9 @@ func stat() gin.HandlerFunc {
 
 		code := fmt.Sprintf("%d", c.Writer.Status())
 		method := c.Request.Method
-		labels := []string{cstats.Service, code, c.FullPath(), method}
+		labels := []string{code, c.FullPath(), method}
 
-		cstats.RequestCounter.WithLabelValues(labels...).Inc()
-		cstats.RequestDuration.WithLabelValues(labels...).Observe(float64(time.Since(start).Seconds()))
+		cstats.RequestDuration.WithLabelValues(labels...).Observe(time.Since(start).Seconds())
 	}
 }
 
@@ -265,11 +264,11 @@ func (rt *Router) Config(r *gin.Engine) {
 		pages.DELETE("/self/token/:id", rt.auth(), rt.user(), rt.deleteToken)
 
 		pages.GET("/users", rt.auth(), rt.user(), rt.perm("/users"), rt.userGets)
-		pages.POST("/users", rt.auth(), rt.admin(), rt.userAddPost)
+		pages.POST("/users", rt.auth(), rt.user(), rt.perm("/users/add"), rt.userAddPost)
 		pages.GET("/user/:id/profile", rt.auth(), rt.userProfileGet)
-		pages.PUT("/user/:id/profile", rt.auth(), rt.admin(), rt.userProfilePut)
-		pages.PUT("/user/:id/password", rt.auth(), rt.admin(), rt.userPasswordPut)
-		pages.DELETE("/user/:id", rt.auth(), rt.admin(), rt.userDel)
+		pages.PUT("/user/:id/profile", rt.auth(), rt.user(), rt.perm("/users/put"), rt.userProfilePut)
+		pages.PUT("/user/:id/password", rt.auth(), rt.user(), rt.perm("/users/put"), rt.userPasswordPut)
+		pages.DELETE("/user/:id", rt.auth(), rt.user(), rt.perm("/users/del"), rt.userDel)
 
 		pages.GET("/metric-views", rt.auth(), rt.metricViewGets)
 		pages.DELETE("/metric-views", rt.auth(), rt.user(), rt.metricViewDel)
@@ -372,6 +371,7 @@ func (rt *Router) Config(r *gin.Engine) {
 		pages.PUT("/busi-group/alert-rule/validate", rt.auth(), rt.user(), rt.perm("/alert-rules/put"), rt.alertRuleValidation)
 		pages.POST("/relabel-test", rt.auth(), rt.user(), rt.relabelTest)
 		pages.POST("/busi-group/:id/alert-rules/clone", rt.auth(), rt.user(), rt.perm("/alert-rules/add"), rt.bgrw(), rt.cloneToMachine)
+		pages.POST("/busi-groups/alert-rules/clones", rt.auth(), rt.user(), rt.perm("/alert-rules/add"), rt.batchAlertRuleClone)
 
 		pages.GET("/busi-groups/recording-rules", rt.auth(), rt.user(), rt.perm("/recording-rules"), rt.recordingRuleGetsByGids)
 		pages.GET("/busi-group/:id/recording-rules", rt.auth(), rt.user(), rt.perm("/recording-rules"), rt.recordingRuleGets)
@@ -389,6 +389,7 @@ func (rt *Router) Config(r *gin.Engine) {
 		pages.PUT("/busi-group/:id/alert-mute/:amid", rt.auth(), rt.user(), rt.perm("/alert-mutes/put"), rt.alertMutePutByFE)
 		pages.GET("/busi-group/:id/alert-mute/:amid", rt.auth(), rt.user(), rt.perm("/alert-mutes"), rt.alertMuteGet)
 		pages.PUT("/busi-group/:id/alert-mutes/fields", rt.auth(), rt.user(), rt.perm("/alert-mutes/put"), rt.bgrw(), rt.alertMutePutFields)
+		pages.POST("/alert-mute-tryrun", rt.auth(), rt.user(), rt.perm("/alert-mutes/add"), rt.alertMuteTryRun)
 
 		pages.GET("/busi-groups/alert-subscribes", rt.auth(), rt.user(), rt.perm("/alert-subscribes"), rt.alertSubscribeGetsByGids)
 		pages.GET("/busi-group/:id/alert-subscribes", rt.auth(), rt.user(), rt.perm("/alert-subscribes"), rt.bgro(), rt.alertSubscribeGets)
@@ -397,21 +398,16 @@ func (rt *Router) Config(r *gin.Engine) {
 		pages.PUT("/busi-group/:id/alert-subscribes", rt.auth(), rt.user(), rt.perm("/alert-subscribes/put"), rt.bgrw(), rt.alertSubscribePut)
 		pages.DELETE("/busi-group/:id/alert-subscribes", rt.auth(), rt.user(), rt.perm("/alert-subscribes/del"), rt.bgrw(), rt.alertSubscribeDel)
 
-		if rt.Center.AnonymousAccess.AlertDetail {
-			pages.GET("/alert-cur-event/:eid", rt.alertCurEventGet)
-			pages.GET("/alert-his-event/:eid", rt.alertHisEventGet)
-			pages.GET("/event-notify-records/:eid", rt.notificationRecordList)
-		} else {
-			pages.GET("/alert-cur-event/:eid", rt.auth(), rt.user(), rt.alertCurEventGet)
-			pages.GET("/alert-his-event/:eid", rt.auth(), rt.user(), rt.alertHisEventGet)
-			pages.GET("/event-notify-records/:eid", rt.auth(), rt.user(), rt.notificationRecordList)
-		}
+		pages.GET("/alert-cur-event/:eid", rt.alertCurEventGet)
+		pages.GET("/alert-his-event/:eid", rt.alertHisEventGet)
+		pages.GET("/event-notify-records/:eid", rt.notificationRecordList)
 
 		// card logic
 		pages.GET("/alert-cur-events/list", rt.auth(), rt.user(), rt.alertCurEventsList)
 		pages.GET("/alert-cur-events/card", rt.auth(), rt.user(), rt.alertCurEventsCard)
 		pages.POST("/alert-cur-events/card/details", rt.auth(), rt.alertCurEventsCardDetails)
 		pages.GET("/alert-his-events/list", rt.auth(), rt.user(), rt.alertHisEventsList)
+		pages.DELETE("/alert-his-events", rt.auth(), rt.admin(), rt.alertHisEventsDelete)
 		pages.DELETE("/alert-cur-events", rt.auth(), rt.user(), rt.perm("/alert-cur-events/del"), rt.alertCurEventDel)
 		pages.GET("/alert-cur-events/stats", rt.auth(), rt.alertCurEventsStatistics)
 
@@ -443,13 +439,13 @@ func (rt *Router) Config(r *gin.Engine) {
 		pages.POST("/datasource/status/update", rt.auth(), rt.admin(), rt.datasourceUpdataStatus)
 		pages.DELETE("/datasource/", rt.auth(), rt.admin(), rt.datasourceDel)
 
-		pages.GET("/roles", rt.auth(), rt.admin(), rt.roleGets)
-		pages.POST("/roles", rt.auth(), rt.admin(), rt.roleAdd)
-		pages.PUT("/roles", rt.auth(), rt.admin(), rt.rolePut)
-		pages.DELETE("/role/:id", rt.auth(), rt.admin(), rt.roleDel)
+		pages.GET("/roles", rt.auth(), rt.user(), rt.perm("/roles"), rt.roleGets)
+		pages.POST("/roles", rt.auth(), rt.user(), rt.perm("/roles/add"), rt.roleAdd)
+		pages.PUT("/roles", rt.auth(), rt.user(), rt.perm("/roles/put"), rt.rolePut)
+		pages.DELETE("/role/:id", rt.auth(), rt.user(), rt.perm("/roles/del"), rt.roleDel)
 
-		pages.GET("/role/:id/ops", rt.auth(), rt.admin(), rt.operationOfRole)
-		pages.PUT("/role/:id/ops", rt.auth(), rt.admin(), rt.roleBindOperation)
+		pages.GET("/role/:id/ops", rt.auth(), rt.user(), rt.perm("/roles"), rt.operationOfRole)
+		pages.PUT("/role/:id/ops", rt.auth(), rt.user(), rt.perm("/roles/put"), rt.roleBindOperation)
 		pages.GET("/operation", rt.operations)
 
 		pages.GET("/notify-tpls", rt.auth(), rt.user(), rt.notifyTplGets)
@@ -471,7 +467,7 @@ func (rt *Router) Config(r *gin.Engine) {
 		pages.GET("/notify-channel", rt.auth(), rt.user(), rt.perm("/help/notification-settings"), rt.notifyChannelGets)
 		pages.PUT("/notify-channel", rt.auth(), rt.admin(), rt.notifyChannelPuts)
 
-		pages.GET("/notify-contact", rt.auth(), rt.user(), rt.perm("/help/notification-settings"), rt.notifyContactGets)
+		pages.GET("/notify-contact", rt.auth(), rt.user(), rt.notifyContactGets)
 		pages.PUT("/notify-contact", rt.auth(), rt.admin(), rt.notifyContactPuts)
 
 		pages.GET("/notify-config", rt.auth(), rt.user(), rt.perm("/help/notification-settings"), rt.notifyConfigGet)
@@ -480,13 +476,20 @@ func (rt *Router) Config(r *gin.Engine) {
 
 		pages.GET("/es-index-pattern", rt.auth(), rt.esIndexPatternGet)
 		pages.GET("/es-index-pattern-list", rt.auth(), rt.esIndexPatternGetList)
-		pages.POST("/es-index-pattern", rt.auth(), rt.admin(), rt.esIndexPatternAdd)
-		pages.PUT("/es-index-pattern", rt.auth(), rt.admin(), rt.esIndexPatternPut)
-		pages.DELETE("/es-index-pattern", rt.auth(), rt.admin(), rt.esIndexPatternDel)
+		pages.POST("/es-index-pattern", rt.auth(), rt.user(), rt.perm("/log/index-patterns/add"), rt.esIndexPatternAdd)
+		pages.PUT("/es-index-pattern", rt.auth(), rt.user(), rt.perm("/log/index-patterns/put"), rt.esIndexPatternPut)
+		pages.DELETE("/es-index-pattern", rt.auth(), rt.user(), rt.perm("/log/index-patterns/del"), rt.esIndexPatternDel)
 
 		pages.GET("/embedded-dashboards", rt.auth(), rt.user(), rt.perm("/embedded-dashboards"), rt.embeddedDashboardsGet)
 		pages.PUT("/embedded-dashboards", rt.auth(), rt.user(), rt.perm("/embedded-dashboards/put"), rt.embeddedDashboardsPut)
 
+		// 获取 embedded-product 列表
+		pages.GET("/embedded-product", rt.auth(), rt.user(), rt.embeddedProductGets)
+		pages.GET("/embedded-product/:id", rt.auth(), rt.user(), rt.embeddedProductGet)
+		pages.POST("/embedded-product", rt.auth(), rt.user(), rt.perm("/embedded-product/add"), rt.embeddedProductAdd)
+		pages.PUT("/embedded-product/:id", rt.auth(), rt.user(), rt.perm("/embedded-product/put"), rt.embeddedProductPut)
+		pages.DELETE("/embedded-product/:id", rt.auth(), rt.user(), rt.perm("/embedded-product/delete"), rt.embeddedProductDelete)
+
 		pages.GET("/user-variable-configs", rt.auth(), rt.user(), rt.perm("/help/variable-configs"), rt.userVariableConfigGets)
 		pages.POST("/user-variable-config", rt.auth(), rt.user(), rt.perm("/help/variable-configs"), rt.userVariableConfigAdd)
 		pages.PUT("/user-variable-config/:id", rt.auth(), rt.user(), rt.perm("/help/variable-configs"), rt.userVariableConfigPut)
@@ -496,22 +499,59 @@ func (rt *Router) Config(r *gin.Engine) {
 		pages.PUT("/config", rt.auth(), rt.admin(), rt.configPutByKey)
 		pages.GET("/site-info", rt.siteInfo)
 
+		// source token 相关路由
+		pages.POST("/source-token", rt.auth(), rt.user(), rt.sourceTokenAdd)
+
 		// for admin api
 		pages.GET("/user/busi-groups", rt.auth(), rt.admin(), rt.userBusiGroupsGets)
 
 		pages.GET("/builtin-components", rt.auth(), rt.user(), rt.builtinComponentsGets)
-		pages.POST("/builtin-components", rt.auth(), rt.user(), rt.perm("/built-in-components/add"), rt.builtinComponentsAdd)
-		pages.PUT("/builtin-components", rt.auth(), rt.user(), rt.perm("/built-in-components/put"), rt.builtinComponentsPut)
-		pages.DELETE("/builtin-components", rt.auth(), rt.user(), rt.perm("/built-in-components/del"), rt.builtinComponentsDel)
+		pages.POST("/builtin-components", rt.auth(), rt.user(), rt.perm("/components/add"), rt.builtinComponentsAdd)
+		pages.PUT("/builtin-components", rt.auth(), rt.user(), rt.perm("/components/put"), rt.builtinComponentsPut)
+		pages.DELETE("/builtin-components", rt.auth(), rt.user(), rt.perm("/components/del"), rt.builtinComponentsDel)
 
 		pages.GET("/builtin-payloads", rt.auth(), rt.user(), rt.builtinPayloadsGets)
 		pages.GET("/builtin-payloads/cates", rt.auth(), rt.user(), rt.builtinPayloadcatesGet)
-		pages.POST("/builtin-payloads", rt.auth(), rt.user(), rt.perm("/built-in-components/add"), rt.builtinPayloadsAdd)
-		pages.GET("/builtin-payload/:id", rt.auth(), rt.user(), rt.perm("/built-in-components"), rt.builtinPayloadGet)
-		pages.PUT("/builtin-payloads", rt.auth(), rt.user(), rt.perm("/built-in-components/put"), rt.builtinPayloadsPut)
-		pages.DELETE("/builtin-payloads", rt.auth(), rt.user(), rt.perm("/built-in-components/del"), rt.builtinPayloadsDel)
+		pages.POST("/builtin-payloads", rt.auth(), rt.user(), rt.perm("/components/add"), rt.builtinPayloadsAdd)
+		pages.GET("/builtin-payload/:id", rt.auth(), rt.user(), rt.perm("/components"), rt.builtinPayloadGet)
+		pages.PUT("/builtin-payloads", rt.auth(), rt.user(), rt.perm("/components/put"), rt.builtinPayloadsPut)
+		pages.DELETE("/builtin-payloads", rt.auth(), rt.user(), rt.perm("/components/del"), rt.builtinPayloadsDel)
 		pages.GET("/builtin-payload", rt.auth(), rt.user(), rt.builtinPayloadsGetByUUIDOrID)
 
+		pages.POST("/message-templates", rt.auth(), rt.user(), rt.perm("/notification-templates/add"), rt.messageTemplatesAdd)
+		pages.DELETE("/message-templates", rt.auth(), rt.user(), rt.perm("/notification-templates/del"), rt.messageTemplatesDel)
+		pages.PUT("/message-template/:id", rt.auth(), rt.user(), rt.perm("/notification-templates/put"), rt.messageTemplatePut)
+		pages.GET("/message-template/:id", rt.auth(), rt.user(), rt.perm("/notification-templates"), rt.messageTemplateGet)
+		pages.GET("/message-templates", rt.auth(), rt.user(), rt.messageTemplatesGet)
+		pages.POST("/events-message", rt.auth(), rt.user(), rt.eventsMessage)
+
+		pages.POST("/notify-rules", rt.auth(), rt.user(), rt.perm("/notification-rules/add"), rt.notifyRulesAdd)
+		pages.DELETE("/notify-rules", rt.auth(), rt.user(), rt.perm("/notification-rules/del"), rt.notifyRulesDel)
+		pages.PUT("/notify-rule/:id", rt.auth(), rt.user(), rt.perm("/notification-rules/put"), rt.notifyRulePut)
+		pages.GET("/notify-rule/:id", rt.auth(), rt.user(), rt.perm("/notification-rules"), rt.notifyRuleGet)
+		pages.GET("/notify-rules", rt.auth(), rt.user(), rt.perm("/notification-rules"), rt.notifyRulesGet)
+		pages.POST("/notify-rule/test", rt.auth(), rt.user(), rt.perm("/notification-rules"), rt.notifyTest)
+		pages.GET("/notify-rule/custom-params", rt.auth(), rt.user(), rt.perm("/notification-rules"), rt.notifyRuleCustomParamsGet)
+		pages.POST("/notify-rule/event-pipelines-tryrun", rt.auth(), rt.user(), rt.perm("/notification-rules/add"), rt.tryRunEventProcessorByNotifyRule)
+
+		// 事件Pipeline相关路由
+		pages.GET("/event-pipelines", rt.auth(), rt.user(), rt.perm("/event-pipelines"), rt.eventPipelinesList)
+		pages.POST("/event-pipeline", rt.auth(), rt.user(), rt.perm("/event-pipelines/add"), rt.addEventPipeline)
+		pages.PUT("/event-pipeline", rt.auth(), rt.user(), rt.perm("/event-pipelines/put"), rt.updateEventPipeline)
+		pages.GET("/event-pipeline/:id", rt.auth(), rt.user(), rt.perm("/event-pipelines"), rt.getEventPipeline)
+		pages.DELETE("/event-pipelines", rt.auth(), rt.user(), rt.perm("/event-pipelines/del"), rt.deleteEventPipelines)
+		pages.POST("/event-pipeline-tryrun", rt.auth(), rt.user(), rt.perm("/event-pipelines"), rt.tryRunEventPipeline)
+		pages.POST("/event-processor-tryrun", rt.auth(), rt.user(), rt.perm("/event-pipelines"), rt.tryRunEventProcessor)
+
+		pages.POST("/notify-channel-configs", rt.auth(), rt.user(), rt.perm("/notification-channels/add"), rt.notifyChannelsAdd)
+		pages.DELETE("/notify-channel-configs", rt.auth(), rt.user(), rt.perm("/notification-channels/del"), rt.notifyChannelsDel)
+		pages.PUT("/notify-channel-config/:id", rt.auth(), rt.user(), rt.perm("/notification-channels/put"), rt.notifyChannelPut)
+		pages.GET("/notify-channel-config/:id", rt.auth(), rt.user(), rt.perm("/notification-channels"), rt.notifyChannelGet)
+		pages.GET("/notify-channel-configs", rt.auth(), rt.user(), rt.perm("/notification-channels"), rt.notifyChannelsGet)
+		pages.GET("/simplified-notify-channel-configs", rt.notifyChannelsGetForNormalUser)
+		pages.GET("/flashduty-channel-list/:id", rt.auth(), rt.user(), rt.flashDutyNotifyChannelsGet)
+		pages.GET("/notify-channel-config", rt.auth(), rt.user(), rt.notifyChannelGetBy)
+		pages.GET("/notify-channel-config/idents", rt.notifyChannelIdentsGet)
 	}
 
 	r.GET("/api/n9e/versions", func(c *gin.Context) {
@@ -554,6 +594,8 @@ func (rt *Router) Config(r *gin.Engine) {
 			service.PUT("/targets/note", rt.targetUpdateNoteByService)
 			service.PUT("/targets/bgid", rt.targetUpdateBgidByService)
 
+			service.POST("/targets-of-host-query", rt.targetsOfHostQuery)
+
 			service.POST("/alert-rules", rt.alertRuleAddByService)
 			service.POST("/alert-rule-add", rt.alertRuleAddOneByService)
 			service.DELETE("/alert-rules", rt.alertRuleDelByService)
@@ -613,6 +655,14 @@ func (rt *Router) Config(r *gin.Engine) {
 			service.POST("/center/heartbeat", rt.heartbeat)
 
 			service.GET("/es-index-pattern-list", rt.esIndexPatternGetList)
+
+			service.GET("/notify-rules", rt.notifyRulesGetByService)
+
+			service.GET("/notify-channels", rt.notifyChannelConfigGets)
+
+			service.GET("/message-templates", rt.messageTemplateGets)
+
+			service.GET("/event-pipelines", rt.eventPipelinesListByService)
 		}
 	}
 

center/router/router_alert_cur_event.go

@@ -1,50 +1,54 @@
 package router
 
 import (
+	"fmt"
 	"net/http"
 	"sort"
 	"strings"
 	"time"
 
 	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/ctx"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 
 	"github.com/gin-gonic/gin"
 	"github.com/toolkits/pkg/ginx"
 )
 
-func parseAggrRules(c *gin.Context) []*models.AggrRule {
-	aggrRules := strings.Split(ginx.QueryStr(c, "rule", ""), "::") // e.g. field:group_name::field:severity::tagkey:ident
-
-	if len(aggrRules) == 0 {
-		ginx.Bomb(http.StatusBadRequest, "rule empty")
+func getUserGroupIds(ctx *gin.Context, rt *Router, myGroups bool) ([]int64, error) {
+	if !myGroups {
+		return nil, nil
 	}
-
-	rules := make([]*models.AggrRule, len(aggrRules))
-	for i := 0; i < len(aggrRules); i++ {
-		pair := strings.Split(aggrRules[i], ":")
-		if len(pair) != 2 {
-			ginx.Bomb(http.StatusBadRequest, "rule invalid")
-		}
-
-		if !(pair[0] == "field" || pair[0] == "tagkey") {
-			ginx.Bomb(http.StatusBadRequest, "rule invalid")
-		}
-
-		rules[i] = &models.AggrRule{
-			Type:  pair[0],
-			Value: pair[1],
-		}
-	}
-
-	return rules
+	me := ctx.MustGet("user").(*models.User)
+	return models.MyGroupIds(rt.Ctx, me.Id)
 }
 
 func (rt *Router) alertCurEventsCard(c *gin.Context) {
 	stime, etime := getTimeRange(c)
-	severity := ginx.QueryInt(c, "severity", -1)
+	severity := strx.IdsInt64ForAPI(ginx.QueryStr(c, "severity", ""), ",")
 	query := ginx.QueryStr(c, "query", "")
+	myGroups := ginx.QueryBool(c, "my_groups", false) // 是否只看自己组，默认false
+
+	var gids []int64
+	var err error
+	if myGroups {
+		gids, err = getUserGroupIds(c, rt, myGroups)
+		ginx.Dangerous(err)
+		if len(gids) == 0 {
+			gids = append(gids, -1)
+		}
+	}
+
+	viewId := ginx.QueryInt64(c, "view_id")
+
+	alertView, err := models.GetAlertAggrViewByViewID(rt.Ctx, viewId)
+	ginx.Dangerous(err)
+
+	if alertView == nil {
+		ginx.Bomb(http.StatusNotFound, "alert aggr view not found")
+	}
+
 	dsIds := queryDatasourceIds(c)
-	rules := parseAggrRules(c)
 
 	prod := ginx.QueryStr(c, "prods", "")
 	if prod == "" {
@@ -61,17 +65,18 @@ func (rt *Router) alertCurEventsCard(c *gin.Context) {
 		cates = strings.Split(cate, ",")
 	}
 
-	bgids, err := GetBusinessGroupIds(c, rt.Ctx, rt.Center.EventHistoryGroupView)
+	bgids, err := GetBusinessGroupIds(c, rt.Ctx, rt.Center.EventHistoryGroupView, myGroups)
 	ginx.Dangerous(err)
 
 	// 最多获取50000个，获取太多也没啥意义
 	list, err := models.AlertCurEventsGet(rt.Ctx, prods, bgids, stime, etime, severity, dsIds,
-		cates, 0, query, 50000, 0)
+		cates, 0, query, 50000, 0, []int64{})
 	ginx.Dangerous(err)
 
 	cardmap := make(map[string]*AlertCard)
 	for _, event := range list {
-		title := event.GenCardTitle(rules)
+		title, err := event.GenCardTitle(alertView.Rule)
+		ginx.Dangerous(err)
 		if _, has := cardmap[title]; has {
 			cardmap[title].Total++
 			cardmap[title].EventIds = append(cardmap[title].EventIds, event.Id)
@@ -86,6 +91,10 @@ func (rt *Router) alertCurEventsCard(c *gin.Context) {
 				Severity: event.Severity,
 			}
 		}
+
+		if cardmap[title].Severity < 1 {
+			cardmap[title].Severity = 3
+		}
 	}
 
 	titles := make([]string, 0, len(cardmap))
@@ -142,11 +151,15 @@ func (rt *Router) alertCurEventsGetByRid(c *gin.Context) {
 // 列表方式，拉取活跃告警
 func (rt *Router) alertCurEventsList(c *gin.Context) {
 	stime, etime := getTimeRange(c)
-	severity := ginx.QueryInt(c, "severity", -1)
+	severity := strx.IdsInt64ForAPI(ginx.QueryStr(c, "severity", ""), ",")
 	query := ginx.QueryStr(c, "query", "")
 	limit := ginx.QueryInt(c, "limit", 20)
+	myGroups := ginx.QueryBool(c, "my_groups", false) // 是否只看自己组，默认false
+
 	dsIds := queryDatasourceIds(c)
 
+	eventIds := strx.IdsInt64ForAPI(ginx.QueryStr(c, "event_ids", ""), ",")
+
 	prod := ginx.QueryStr(c, "prods", "")
 	if prod == "" {
 		prod = ginx.QueryStr(c, "rule_prods", "")
@@ -165,18 +178,19 @@ func (rt *Router) alertCurEventsList(c *gin.Context) {
 
 	ruleId := ginx.QueryInt64(c, "rid", 0)
 
-	bgids, err := GetBusinessGroupIds(c, rt.Ctx, rt.Center.EventHistoryGroupView)
+	bgids, err := GetBusinessGroupIds(c, rt.Ctx, rt.Center.EventHistoryGroupView, myGroups)
 	ginx.Dangerous(err)
 
 	total, err := models.AlertCurEventTotal(rt.Ctx, prods, bgids, stime, etime, severity, dsIds,
-		cates, ruleId, query)
+		cates, ruleId, query, eventIds)
 	ginx.Dangerous(err)
 
 	list, err := models.AlertCurEventsGet(rt.Ctx, prods, bgids, stime, etime, severity, dsIds,
-		cates, ruleId, query, limit, ginx.Offset(c, limit))
+		cates, ruleId, query, limit, ginx.Offset(c, limit), eventIds)
 	ginx.Dangerous(err)
 
 	cache := make(map[int64]*models.UserGroup)
+
 	for i := 0; i < len(list); i++ {
 		list[i].FillNotifyGroups(rt.Ctx, cache)
 	}
@@ -218,24 +232,68 @@ func (rt *Router) checkCurEventBusiGroupRWPermission(c *gin.Context, ids []int64
 
 func (rt *Router) alertCurEventGet(c *gin.Context) {
 	eid := ginx.UrlParamInt64(c, "eid")
-	event, err := models.AlertCurEventGetById(rt.Ctx, eid)
-	ginx.Dangerous(err)
+	event, err := GetCurEventDetail(rt.Ctx, eid)
 
-	if event == nil {
-		ginx.Bomb(404, "No such active event")
-	}
-
-	if !rt.Center.AnonymousAccess.AlertDetail && rt.Center.EventHistoryGroupView {
+	hasPermission := HasPermission(rt.Ctx, c, "event", fmt.Sprintf("%d", eid), rt.Center.AnonymousAccess.AlertDetail)
+	if !hasPermission {
+		rt.auth()(c)
+		rt.user()(c)
 		rt.bgroCheck(c, event.GroupId)
 	}
 
-	ruleConfig, needReset := models.FillRuleConfigTplName(rt.Ctx, event.RuleConfig)
+	ginx.NewRender(c).Data(event, err)
+}
+
+func GetCurEventDetail(ctx *ctx.Context, eid int64) (*models.AlertCurEvent, error) {
+	event, err := models.AlertCurEventGetById(ctx, eid)
+	if err != nil {
+		return nil, err
+	}
+
+	if event == nil {
+		return nil, fmt.Errorf("no such active event")
+	}
+
+	ruleConfig, needReset := models.FillRuleConfigTplName(ctx, event.RuleConfig)
 	if needReset {
 		event.RuleConfigJson = ruleConfig
 	}
 
 	event.LastEvalTime = event.TriggerTime
-	ginx.NewRender(c).Data(event, nil)
+	event.NotifyVersion, err = GetEventNotifyVersion(ctx, event.RuleId, event.NotifyRuleIds)
+	ginx.Dangerous(err)
+
+	event.NotifyRules, err = GetEventNorifyRuleNames(ctx, event.NotifyRuleIds)
+	return event, err
+}
+
+func GetEventNorifyRuleNames(ctx *ctx.Context, notifyRuleIds []int64) ([]*models.EventNotifyRule, error) {
+	notifyRuleNames := make([]*models.EventNotifyRule, 0)
+	notifyRules, err := models.NotifyRulesGet(ctx, "id in ?", notifyRuleIds)
+	if err != nil {
+		return nil, err
+	}
+
+	for _, notifyRule := range notifyRules {
+		notifyRuleNames = append(notifyRuleNames, &models.EventNotifyRule{
+			Id:   notifyRule.ID,
+			Name: notifyRule.Name,
+		})
+	}
+	return notifyRuleNames, nil
+}
+
+func GetEventNotifyVersion(ctx *ctx.Context, ruleId int64, notifyRuleIds []int64) (int, error) {
+	if len(notifyRuleIds) != 0 {
+		// 如果存在 notify_rule_ids，则认为使用新的告警通知方式
+		return 1, nil
+	}
+
+	rule, err := models.AlertRuleGetById(ctx, ruleId)
+	if err != nil {
+		return 0, err
+	}
+	return rule.NotifyVersion, nil
 }
 
 func (rt *Router) alertCurEventsStatistics(c *gin.Context) {

center/router/router_alert_his_event.go

@@ -2,6 +2,7 @@ package router
 
 import (
 	"fmt"
+	"net/http"
 	"strings"
 	"time"
 
@@ -10,6 +11,7 @@ import (
 
 	"github.com/gin-gonic/gin"
 	"github.com/toolkits/pkg/ginx"
+	"github.com/toolkits/pkg/logger"
 	"golang.org/x/exp/slices"
 )
 
@@ -56,7 +58,7 @@ func (rt *Router) alertHisEventsList(c *gin.Context) {
 
 	ruleId := ginx.QueryInt64(c, "rid", 0)
 
-	bgids, err := GetBusinessGroupIds(c, rt.Ctx, rt.Center.EventHistoryGroupView)
+	bgids, err := GetBusinessGroupIds(c, rt.Ctx, rt.Center.EventHistoryGroupView, false)
 	ginx.Dangerous(err)
 
 	total, err := models.AlertHisEventTotal(rt.Ctx, prods, bgids, stime, etime, severity,
@@ -78,16 +80,56 @@ func (rt *Router) alertHisEventsList(c *gin.Context) {
 	}, nil)
 }
 
+type alertHisEventsDeleteForm struct {
+	Severities []int `json:"severities"`
+	Timestamp  int64 `json:"timestamp" binding:"required"`
+}
+
+func (rt *Router) alertHisEventsDelete(c *gin.Context) {
+	var f alertHisEventsDeleteForm
+	ginx.BindJSON(c, &f)
+	// 校验
+	if f.Timestamp == 0 {
+		ginx.Bomb(http.StatusBadRequest, "timestamp parameter is required")
+		return
+	}
+
+	user := c.MustGet("user").(*models.User)
+
+	// 启动后台清理任务
+	go func() {
+		limit := 100
+		for {
+			n, err := models.AlertHisEventBatchDelete(rt.Ctx, f.Timestamp, f.Severities, limit)
+			if err != nil {
+				logger.Errorf("Failed to delete alert history events: operator=%s, timestamp=%d, severities=%v, error=%v",
+					user.Username, f.Timestamp, f.Severities, err)
+				break
+			}
+			logger.Debugf("Successfully deleted alert history events: operator=%s, timestamp=%d, severities=%v, deleted=%d",
+				user.Username, f.Timestamp, f.Severities, n)
+			if n < int64(limit) {
+				break // 已经删完
+			}
+
+			time.Sleep(100 * time.Millisecond) // 防止锁表
+		}
+	}()
+	ginx.NewRender(c).Message("Alert history events deletion started")
+}
+
 func (rt *Router) alertHisEventGet(c *gin.Context) {
 	eid := ginx.UrlParamInt64(c, "eid")
 	event, err := models.AlertHisEventGetById(rt.Ctx, eid)
 	ginx.Dangerous(err)
-
 	if event == nil {
 		ginx.Bomb(404, "No such alert event")
 	}
 
-	if !rt.Center.AnonymousAccess.AlertDetail && rt.Center.EventHistoryGroupView {
+	hasPermission := HasPermission(rt.Ctx, c, "event", fmt.Sprintf("%d", eid), rt.Center.AnonymousAccess.AlertDetail)
+	if !hasPermission {
+		rt.auth()(c)
+		rt.user()(c)
 		rt.bgroCheck(c, event.GroupId)
 	}
 
@@ -96,46 +138,54 @@ func (rt *Router) alertHisEventGet(c *gin.Context) {
 		event.RuleConfigJson = ruleConfig
 	}
 
+	event.NotifyVersion, err = GetEventNotifyVersion(rt.Ctx, event.RuleId, event.NotifyRuleIds)
+	ginx.Dangerous(err)
+
+	event.NotifyRules, err = GetEventNorifyRuleNames(rt.Ctx, event.NotifyRuleIds)
 	ginx.NewRender(c).Data(event, err)
 }
 
-func GetBusinessGroupIds(c *gin.Context, ctx *ctx.Context, eventHistoryGroupView bool) ([]int64, error) {
+func GetBusinessGroupIds(c *gin.Context, ctx *ctx.Context, onlySelfGroupView bool, myGroups bool) ([]int64, error) {
 	bgid := ginx.QueryInt64(c, "bgid", 0)
 	var bgids []int64
 
-	if !eventHistoryGroupView || strings.HasPrefix(c.Request.URL.Path, "/v1") {
+	if strings.HasPrefix(c.Request.URL.Path, "/v1") {
+		// 如果请求路径以 /v1 开头，不查询用户信息
 		if bgid > 0 {
 			return []int64{bgid}, nil
 		}
+
 		return bgids, nil
 	}
 
 	user := c.MustGet("user").(*models.User)
-	if user.IsAdmin() {
+	if myGroups || (onlySelfGroupView && !user.IsAdmin()) {
+		// 1. 页面上勾选了我的业务组，需要查询用户所属的业务组
+		// 2. 如果 onlySelfGroupView 为 true，表示只允许查询用户所属的业务组
+		bussGroupIds, err := models.MyBusiGroupIds(ctx, user.Id)
+		if err != nil {
+			return nil, err
+		}
+
+		if len(bussGroupIds) == 0 {
+			// 如果没查到用户属于任何业务组，需要返回一个0，否则会导致查询到全部告警历史
+			return []int64{0}, nil
+		}
+
 		if bgid > 0 {
+			if !slices.Contains(bussGroupIds, bgid) && !user.IsAdmin() {
+				return nil, fmt.Errorf("business group ID not allowed")
+			}
+
 			return []int64{bgid}, nil
 		}
-		return bgids, nil
-	}
 
-	bussGroupIds, err := models.MyBusiGroupIds(ctx, user.Id)
-	if err != nil {
-		return nil, err
-	}
-
-	if len(bussGroupIds) == 0 {
-		// 如果没查到用户属于任何业务组，需要返回一个0，否则会导致查询到全部告警历史
-		return []int64{0}, nil
-	}
-
-	if bgid > 0 && !slices.Contains(bussGroupIds, bgid) {
-		return nil, fmt.Errorf("business group ID not allowed")
+		return bussGroupIds, nil
 	}
 
 	if bgid > 0 {
-		// Pass filter parameters, priority to use
 		return []int64{bgid}, nil
 	}
 
-	return bussGroupIds, nil
+	return bgids, nil
 }

center/router/router_alert_rule.go

@@ -12,6 +12,7 @@ import (
 	"gopkg.in/yaml.v2"
 
 	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 	"github.com/ccfos/nightingale/v6/pushgw/pconf"
 	"github.com/ccfos/nightingale/v6/pushgw/writer"
 
@@ -20,7 +21,6 @@ import (
 	"github.com/prometheus/prometheus/prompb"
 	"github.com/toolkits/pkg/ginx"
 	"github.com/toolkits/pkg/i18n"
-	"github.com/toolkits/pkg/str"
 )
 
 type AlertRuleModifyHookFunc func(ar *models.AlertRule)
@@ -52,7 +52,7 @@ func getAlertCueEventTimeRange(c *gin.Context) (stime, etime int64) {
 }
 
 func (rt *Router) alertRuleGetsByGids(c *gin.Context) {
-	gids := str.IdsInt64(ginx.QueryStr(c, "gids", ""), ",")
+	gids := strx.IdsInt64ForAPI(ginx.QueryStr(c, "gids", ""), ",")
 	if len(gids) > 0 {
 		for _, gid := range gids {
 			rt.bgroCheck(c, gid)
@@ -428,7 +428,16 @@ func (rt *Router) alertRulePutFields(c *gin.Context) {
 		}
 
 		for k, v := range f.Fields {
-			ginx.Dangerous(ar.UpdateColumn(rt.Ctx, k, v))
+			// 检查 v 是否为各种切片类型
+			switch v.(type) {
+			case []interface{}, []int64, []int, []string:
+				// 将切片转换为 JSON 字符串
+				bytes, err := json.Marshal(v)
+				ginx.Dangerous(err)
+				ginx.Dangerous(ar.UpdateColumn(rt.Ctx, k, string(bytes)))
+			default:
+				ginx.Dangerous(ar.UpdateColumn(rt.Ctx, k, v))
+			}
 		}
 	}
 
@@ -678,3 +687,49 @@ func (rt *Router) cloneToMachine(c *gin.Context) {
 
 	ginx.NewRender(c).Data(reterr, models.InsertAlertRule(rt.Ctx, newRules))
 }
+
+type alertBatchCloneForm struct {
+	RuleIds []int64 `json:"rule_ids"`
+	Bgids   []int64 `json:"bgids"`
+}
+
+// 批量克隆告警规则
+func (rt *Router) batchAlertRuleClone(c *gin.Context) {
+	me := c.MustGet("user").(*models.User)
+
+	var f alertBatchCloneForm
+	ginx.BindJSON(c, &f)
+
+	// 校验 bgids 操作权限
+	for _, bgid := range f.Bgids {
+		rt.bgrwCheck(c, bgid)
+	}
+
+	reterr := make(map[string]string, len(f.RuleIds))
+	lang := c.GetHeader("X-Language")
+
+	for _, arid := range f.RuleIds {
+		ar, err := models.AlertRuleGetById(rt.Ctx, arid)
+		for _, bgid := range f.Bgids {
+			// 为了让 bgid 和 arid 对应，将上面的 err 放到这里处理
+			if err != nil {
+				reterr[fmt.Sprintf("%d-%d", arid, bgid)] = i18n.Sprintf(lang, err.Error())
+				continue
+			}
+
+			if ar == nil {
+				reterr[fmt.Sprintf("%d-%d", arid, bgid)] = i18n.Sprintf(lang, "alert rule not found")
+				continue
+			}
+
+			newAr := ar.Clone(me.Username, bgid)
+			err = newAr.Add(rt.Ctx)
+			if err != nil {
+				reterr[fmt.Sprintf("%d-%d", arid, bgid)] = i18n.Sprintf(lang, err.Error())
+				continue
+			}
+		}
+	}
+
+	ginx.NewRender(c).Data(reterr, nil)
+}

center/router/router_alert_subscribe.go

@@ -5,10 +5,10 @@ import (
 	"time"
 
 	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 
 	"github.com/gin-gonic/gin"
 	"github.com/toolkits/pkg/ginx"
-	"github.com/toolkits/pkg/str"
 )
 
 // Return all, front-end search and paging
@@ -31,7 +31,7 @@ func (rt *Router) alertSubscribeGets(c *gin.Context) {
 }
 
 func (rt *Router) alertSubscribeGetsByGids(c *gin.Context) {
-	gids := str.IdsInt64(ginx.QueryStr(c, "gids", ""), ",")
+	gids := strx.IdsInt64ForAPI(ginx.QueryStr(c, "gids", ""), ",")
 	if len(gids) > 0 {
 		for _, gid := range gids {
 			rt.bgroCheck(c, gid)
@@ -141,6 +141,7 @@ func (rt *Router) alertSubscribePut(c *gin.Context) {
 			"extra_config",
 			"busi_groups",
 			"note",
+			"notify_rule_ids",
 		))
 	}
 

center/router/router_board.go

@@ -6,11 +6,11 @@ import (
 	"time"
 
 	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 
 	"github.com/gin-gonic/gin"
 	"github.com/toolkits/pkg/ginx"
 	"github.com/toolkits/pkg/i18n"
-	"github.com/toolkits/pkg/str"
 )
 
 type boardForm struct {
@@ -51,9 +51,14 @@ func (rt *Router) boardAdd(c *gin.Context) {
 
 func (rt *Router) boardGet(c *gin.Context) {
 	bid := ginx.UrlParamStr(c, "bid")
-	board, err := models.BoardGet(rt.Ctx, "id = ? or ident = ?", bid, bid)
+	board, err := models.BoardGet(rt.Ctx, "ident = ?", bid)
 	ginx.Dangerous(err)
 
+	if board == nil {
+		board, err = models.BoardGet(rt.Ctx, "id = ?", bid)
+		ginx.Dangerous(err)
+	}
+
 	if board == nil {
 		ginx.Bomb(http.StatusNotFound, "No such dashboard")
 	}
@@ -96,7 +101,7 @@ func (rt *Router) boardGet(c *gin.Context) {
 
 // 根据 bids 参数，获取多个 board
 func (rt *Router) boardGetsByBids(c *gin.Context) {
-	bids := str.IdsInt64(ginx.QueryStr(c, "bids", ""), ",")
+	bids := strx.IdsInt64ForAPI(ginx.QueryStr(c, "bids", ""), ",")
 	boards, err := models.BoardGetsByBids(rt.Ctx, bids)
 	ginx.Dangerous(err)
 	ginx.NewRender(c).Data(boards, err)
@@ -265,7 +270,7 @@ func (rt *Router) publicBoardGets(c *gin.Context) {
 }
 
 func (rt *Router) boardGetsByGids(c *gin.Context) {
-	gids := str.IdsInt64(ginx.QueryStr(c, "gids", ""), ",")
+	gids := strx.IdsInt64ForAPI(ginx.QueryStr(c, "gids", ""), ",")
 	query := ginx.QueryStr(c, "query", "")
 
 	if len(gids) > 0 {

center/router/router_builtin_metric_filter.go

@@ -57,7 +57,7 @@ func (rt *Router) metricFilterDel(c *gin.Context) {
 			ginx.Dangerous(err)
 
 			if !HasPerm(gids, old.GroupsPerm, true) {
-				ginx.NewRender(c).Message("no permission")
+				ginx.NewRender(c).Message("forbidden")
 				return
 			}
 		}
@@ -79,7 +79,7 @@ func (rt *Router) metricFilterPut(c *gin.Context) {
 		ginx.Dangerous(err)
 
 		if !HasPerm(gids, old.GroupsPerm, true) {
-			ginx.NewRender(c).Message("no permission")
+			ginx.NewRender(c).Message("forbidden")
 			return
 		}
 	}

center/router/router_builtin_metrics.go

@@ -86,15 +86,11 @@ func (rt *Router) builtinMetricsDel(c *gin.Context) {
 func (rt *Router) builtinMetricsDefaultTypes(c *gin.Context) {
 	lst := []string{
 		"Linux",
+		"Procstat",
 		"cAdvisor",
 		"Ping",
 		"MySQL",
-		"Redis",
-		"Kafka",
-		"Elasticsearch",
-		"PostgreSQL",
-		"MongoDB",
-		"Memcached",
+		"ClickHouse",
 	}
 	ginx.NewRender(c).Data(lst, nil)
 }
@@ -102,29 +98,10 @@ func (rt *Router) builtinMetricsDefaultTypes(c *gin.Context) {
 func (rt *Router) builtinMetricsTypes(c *gin.Context) {
 	collector := ginx.QueryStr(c, "collector", "")
 	query := ginx.QueryStr(c, "query", "")
-	disabled := ginx.QueryInt(c, "disabled", -1)
 	lang := c.GetHeader("X-Language")
 
 	metricTypeList, err := models.BuiltinMetricTypes(rt.Ctx, lang, collector, query)
-	ginx.Dangerous(err)
-
-	componentList, err := models.BuiltinComponentGets(rt.Ctx, "", disabled)
-	ginx.Dangerous(err)
-
-	// 创建一个 map 来存储 componentList 中的类型
-	componentTypes := make(map[string]struct{})
-	for _, comp := range componentList {
-		componentTypes[comp.Ident] = struct{}{}
-	}
-
-	filteredMetricTypeList := make([]string, 0)
-	for _, metricType := range metricTypeList {
-		if _, exists := componentTypes[metricType]; exists {
-			filteredMetricTypeList = append(filteredMetricTypeList, metricType)
-		}
-	}
-
-	ginx.NewRender(c).Data(filteredMetricTypeList, nil)
+	ginx.NewRender(c).Data(metricTypeList, err)
 }
 
 func (rt *Router) builtinMetricsCollectors(c *gin.Context) {

center/router/router_busi_group.go

@@ -4,11 +4,11 @@ import (
 	"net/http"
 
 	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 
 	"github.com/gin-gonic/gin"
 	"github.com/toolkits/pkg/ginx"
 	"github.com/toolkits/pkg/logger"
-	"github.com/toolkits/pkg/str"
 )
 
 type busiGroupForm struct {
@@ -131,7 +131,7 @@ func (rt *Router) busiGroupGetsByService(c *gin.Context) {
 // 这个接口只有在活跃告警页面才调用，获取各个BG的活跃告警数量
 func (rt *Router) busiGroupAlertingsGets(c *gin.Context) {
 	ids := ginx.QueryStr(c, "ids", "")
-	ret, err := models.AlertNumbers(rt.Ctx, str.IdsInt64(ids))
+	ret, err := models.AlertNumbers(rt.Ctx, strx.IdsInt64ForAPI(ids))
 	ginx.NewRender(c).Data(ret, err)
 }
 
@@ -142,7 +142,7 @@ func (rt *Router) busiGroupGet(c *gin.Context) {
 }
 
 func (rt *Router) busiGroupsGetTags(c *gin.Context) {
-	bgids := str.IdsInt64(ginx.QueryStr(c, "gids", ""), ",")
+	bgids := strx.IdsInt64ForAPI(ginx.QueryStr(c, "gids", ""), ",")
 	targetIdents, err := models.TargetIndentsGetByBgids(rt.Ctx, bgids)
 	ginx.Dangerous(err)
 	tags, err := models.TargetGetTags(rt.Ctx, targetIdents, true, "busigroup")

center/router/router_chart_share.go

@@ -4,15 +4,15 @@ import (
 	"time"
 
 	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 
 	"github.com/gin-gonic/gin"
 	"github.com/toolkits/pkg/ginx"
-	"github.com/toolkits/pkg/str"
 )
 
 func (rt *Router) chartShareGets(c *gin.Context) {
 	ids := ginx.QueryStr(c, "ids", "")
-	lst, err := models.ChartShareGetsByIds(rt.Ctx, str.IdsInt64(ids, ","))
+	lst, err := models.ChartShareGetsByIds(rt.Ctx, strx.IdsInt64ForAPI(ids, ","))
 	ginx.NewRender(c).Data(lst, err)
 }
 

center/router/router_datasource.go

@@ -57,15 +57,21 @@ func (rt *Router) datasourceBriefs(c *gin.Context) {
 
 	for _, item := range list {
 		item.AuthJson.BasicAuthPassword = ""
-		if item.PluginType != models.PROMETHEUS {
-			item.SettingsJson = nil
-		} else {
+		if item.PluginType == models.PROMETHEUS {
 			for k, v := range item.SettingsJson {
 				if strings.HasPrefix(k, "prometheus.") {
 					item.SettingsJson[strings.TrimPrefix(k, "prometheus.")] = v
 					delete(item.SettingsJson, k)
 				}
 			}
+		} else if item.PluginType == "cloudwatch" {
+			for k := range item.SettingsJson {
+				if !strings.Contains(k, "region") {
+					delete(item.SettingsJson, k)
+				}
+			}
+		} else {
+			item.SettingsJson = nil
 		}
 		dss = append(dss, item)
 	}
@@ -117,7 +123,7 @@ func (rt *Router) datasourceUpsert(c *gin.Context) {
 		}
 		err = req.Add(rt.Ctx)
 	} else {
-		err = req.Update(rt.Ctx, "name", "description", "cluster_name", "settings", "http", "auth", "updated_by", "updated_at", "is_default")
+		err = req.Update(rt.Ctx, "name", "identifier", "description", "cluster_name", "settings", "http", "auth", "updated_by", "updated_at", "is_default")
 	}
 
 	Render(c, nil, err)
@@ -142,11 +148,12 @@ func DatasourceCheck(ds models.Datasource) error {
 		},
 	}
 
+	ds.HTTPJson.Url = strings.TrimRight(ds.HTTPJson.Url, "/")
 	var fullURL string
 	req, err := ds.HTTPJson.NewReq(&fullURL)
 	if err != nil {
 		logger.Errorf("Error creating request: %v", err)
-		return fmt.Errorf("request urls:%v failed", ds.HTTPJson.GetUrls())
+		return fmt.Errorf("request urls:%v failed: %v", ds.HTTPJson.GetUrls(), err)
 	}
 
 	if ds.PluginType == models.PROMETHEUS {
@@ -162,14 +169,14 @@ func DatasourceCheck(ds models.Datasource) error {
 		req, err = http.NewRequest("GET", fullURL, nil)
 		if err != nil {
 			logger.Errorf("Error creating request: %v", err)
-			return fmt.Errorf("request url:%s failed", fullURL)
+			return fmt.Errorf("request url:%s failed: %v", fullURL, err)
 		}
 	} else if ds.PluginType == models.TDENGINE {
 		fullURL = fmt.Sprintf("%s/rest/sql", ds.HTTPJson.Url)
 		req, err = http.NewRequest("POST", fullURL, strings.NewReader("show databases"))
 		if err != nil {
 			logger.Errorf("Error creating request: %v", err)
-			return fmt.Errorf("request url:%s failed", fullURL)
+			return fmt.Errorf("request url:%s failed: %v", fullURL, err)
 		}
 	}
 
@@ -181,7 +188,7 @@ func DatasourceCheck(ds models.Datasource) error {
 		req, err = http.NewRequest("GET", fullURL, nil)
 		if err != nil {
 			logger.Errorf("Error creating request: %v", err)
-			return fmt.Errorf("request url:%s failed", fullURL)
+			return fmt.Errorf("request url:%s failed: %v", fullURL, err)
 		}
 	}
 
@@ -196,7 +203,7 @@ func DatasourceCheck(ds models.Datasource) error {
 	resp, err := client.Do(req)
 	if err != nil {
 		logger.Errorf("Error making request: %v\n", err)
-		return fmt.Errorf("request url:%s failed", fullURL)
+		return fmt.Errorf("request url:%s failed: %v", fullURL, err)
 	}
 	defer resp.Body.Close()
 

center/router/router_embedded.go

@@ -0,0 +1,141 @@
+package router
+
+import (
+	"time"
+
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/ctx"
+
+	"github.com/gin-gonic/gin"
+	"github.com/toolkits/pkg/ginx"
+)
+
+func (rt *Router) embeddedProductGets(c *gin.Context) {
+	products, err := models.EmbeddedProductGets(rt.Ctx)
+	ginx.Dangerous(err)
+	// 获取当前用户可访问的Group ID 列表
+	me := c.MustGet("user").(*models.User)
+
+	if me.IsAdmin() {
+		ginx.NewRender(c).Data(products, err)
+		return
+	}
+
+	gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+	bgSet := make(map[int64]struct{}, len(gids))
+	for _, id := range gids {
+		bgSet[id] = struct{}{}
+	}
+
+	// 过滤出公开或有权限访问的私有 product link
+	var result []*models.EmbeddedProduct
+	for _, product := range products {
+		if !product.IsPrivate {
+			result = append(result, product)
+			continue
+		}
+
+		for _, tid := range product.TeamIDs {
+			if _, ok := bgSet[tid]; ok {
+				result = append(result, product)
+				break
+			}
+		}
+	}
+
+	ginx.NewRender(c).Data(result, err)
+}
+
+func (rt *Router) embeddedProductGet(c *gin.Context) {
+	id := ginx.UrlParamInt64(c, "id")
+	if id <= 0 {
+		ginx.Bomb(400, "invalid id")
+	}
+
+	data, err := models.GetEmbeddedProductByID(rt.Ctx, id)
+	ginx.Dangerous(err)
+
+	me := c.MustGet("user").(*models.User)
+	hashPermission, err := hasEmbeddedProductAccess(rt.Ctx, me, data)
+	ginx.Dangerous(err)
+
+	if !hashPermission {
+		ginx.Bomb(403, "forbidden")
+	}
+
+	ginx.NewRender(c).Data(data, nil)
+}
+
+func (rt *Router) embeddedProductAdd(c *gin.Context) {
+	var eps []models.EmbeddedProduct
+	ginx.BindJSON(c, &eps)
+
+	me := c.MustGet("user").(*models.User)
+
+	for i := range eps {
+		eps[i].CreateBy = me.Nickname
+		eps[i].UpdateBy = me.Nickname
+	}
+
+	err := models.AddEmbeddedProduct(rt.Ctx, eps)
+	ginx.NewRender(c).Message(err)
+}
+
+func (rt *Router) embeddedProductPut(c *gin.Context) {
+	var ep models.EmbeddedProduct
+	id := ginx.UrlParamInt64(c, "id")
+	ginx.BindJSON(c, &ep)
+
+	if id <= 0 {
+		ginx.Bomb(400, "invalid id")
+	}
+
+	oldProduct, err := models.GetEmbeddedProductByID(rt.Ctx, id)
+	ginx.Dangerous(err)
+	me := c.MustGet("user").(*models.User)
+
+	now := time.Now().Unix()
+	oldProduct.Name = ep.Name
+	oldProduct.URL = ep.URL
+	oldProduct.IsPrivate = ep.IsPrivate
+	oldProduct.TeamIDs = ep.TeamIDs
+	oldProduct.UpdateBy = me.Username
+	oldProduct.UpdateAt = now
+
+	err = models.UpdateEmbeddedProduct(rt.Ctx, oldProduct)
+	ginx.NewRender(c).Message(err)
+}
+
+func (rt *Router) embeddedProductDelete(c *gin.Context) {
+	id := ginx.UrlParamInt64(c, "id")
+	if id <= 0 {
+		ginx.Bomb(400, "invalid id")
+	}
+
+	err := models.DeleteEmbeddedProduct(rt.Ctx, id)
+	ginx.NewRender(c).Message(err)
+}
+
+func hasEmbeddedProductAccess(ctx *ctx.Context, user *models.User, ep *models.EmbeddedProduct) (bool, error) {
+	if user.IsAdmin() || !ep.IsPrivate {
+		return true, nil
+	}
+
+	gids, err := models.MyGroupIds(ctx, user.Id)
+	if err != nil {
+		return false, err
+	}
+
+	groupSet := make(map[int64]struct{}, len(gids))
+	for _, gid := range gids {
+		groupSet[gid] = struct{}{}
+	}
+
+	for _, tid := range ep.TeamIDs {
+		if _, ok := groupSet[tid]; ok {
+			return true, nil
+		}
+	}
+
+	return false, nil
+}

center/router/router_event_pipeline.go

@@ -0,0 +1,245 @@
+package router
+
+import (
+	"net/http"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/models"
+
+	"github.com/gin-gonic/gin"
+	"github.com/toolkits/pkg/ginx"
+)
+
+// 获取事件Pipeline列表
+func (rt *Router) eventPipelinesList(c *gin.Context) {
+	me := c.MustGet("user").(*models.User)
+	pipelines, err := models.ListEventPipelines(rt.Ctx)
+	ginx.Dangerous(err)
+
+	allTids := make([]int64, 0)
+	for _, pipeline := range pipelines {
+		allTids = append(allTids, pipeline.TeamIds...)
+	}
+	ugMap, err := models.UserGroupIdAndNameMap(rt.Ctx, allTids)
+	ginx.Dangerous(err)
+	for _, pipeline := range pipelines {
+		for _, tid := range pipeline.TeamIds {
+			pipeline.TeamNames = append(pipeline.TeamNames, ugMap[tid])
+		}
+	}
+
+	gids, err := models.MyGroupIdsMap(rt.Ctx, me.Id)
+	ginx.Dangerous(err)
+
+	if me.IsAdmin() {
+		ginx.NewRender(c).Data(pipelines, nil)
+		return
+	}
+
+	res := make([]*models.EventPipeline, 0)
+	for _, pipeline := range pipelines {
+		for _, tid := range pipeline.TeamIds {
+			if _, ok := gids[tid]; ok {
+				res = append(res, pipeline)
+				break
+			}
+		}
+	}
+
+	ginx.NewRender(c).Data(res, nil)
+}
+
+// 获取单个事件Pipeline详情
+func (rt *Router) getEventPipeline(c *gin.Context) {
+	me := c.MustGet("user").(*models.User)
+	id := ginx.UrlParamInt64(c, "id")
+	pipeline, err := models.GetEventPipeline(rt.Ctx, id)
+	ginx.Dangerous(err)
+	ginx.Dangerous(me.CheckGroupPermission(rt.Ctx, pipeline.TeamIds))
+
+	err = pipeline.FillTeamNames(rt.Ctx)
+	ginx.Dangerous(err)
+
+	ginx.NewRender(c).Data(pipeline, nil)
+}
+
+// 创建事件Pipeline
+func (rt *Router) addEventPipeline(c *gin.Context) {
+	var pipeline models.EventPipeline
+	ginx.BindJSON(c, &pipeline)
+
+	user := c.MustGet("user").(*models.User)
+	now := time.Now().Unix()
+	pipeline.CreateBy = user.Username
+	pipeline.CreateAt = now
+	pipeline.UpdateAt = now
+	pipeline.UpdateBy = user.Username
+
+	err := pipeline.Verify()
+	if err != nil {
+		ginx.Bomb(http.StatusBadRequest, err.Error())
+	}
+
+	ginx.Dangerous(user.CheckGroupPermission(rt.Ctx, pipeline.TeamIds))
+	err = models.CreateEventPipeline(rt.Ctx, &pipeline)
+	ginx.NewRender(c).Message(err)
+}
+
+// 更新事件Pipeline
+func (rt *Router) updateEventPipeline(c *gin.Context) {
+	var f models.EventPipeline
+	ginx.BindJSON(c, &f)
+
+	me := c.MustGet("user").(*models.User)
+	f.UpdateBy = me.Username
+	f.UpdateAt = time.Now().Unix()
+
+	pipeline, err := models.GetEventPipeline(rt.Ctx, f.ID)
+	if err != nil {
+		ginx.Bomb(http.StatusNotFound, "No such event pipeline")
+	}
+	ginx.Dangerous(me.CheckGroupPermission(rt.Ctx, pipeline.TeamIds))
+
+	ginx.NewRender(c).Message(pipeline.Update(rt.Ctx, &f))
+}
+
+// 删除事件Pipeline
+func (rt *Router) deleteEventPipelines(c *gin.Context) {
+	var f struct {
+		Ids []int64 `json:"ids"`
+	}
+	ginx.BindJSON(c, &f)
+
+	if len(f.Ids) == 0 {
+		ginx.Bomb(http.StatusBadRequest, "ids required")
+	}
+
+	me := c.MustGet("user").(*models.User)
+	for _, id := range f.Ids {
+		pipeline, err := models.GetEventPipeline(rt.Ctx, id)
+		ginx.Dangerous(err)
+		ginx.Dangerous(me.CheckGroupPermission(rt.Ctx, pipeline.TeamIds))
+	}
+
+	err := models.DeleteEventPipelines(rt.Ctx, f.Ids)
+	ginx.NewRender(c).Message(err)
+}
+
+// 测试事件Pipeline
+func (rt *Router) tryRunEventPipeline(c *gin.Context) {
+	var f struct {
+		EventId        int64                `json:"event_id"`
+		PipelineConfig models.EventPipeline `json:"pipeline_config"`
+	}
+	ginx.BindJSON(c, &f)
+
+	hisEvent, err := models.AlertHisEventGetById(rt.Ctx, f.EventId)
+	if err != nil || hisEvent == nil {
+		ginx.Bomb(http.StatusBadRequest, "event not found")
+	}
+	event := hisEvent.ToCur()
+
+	for _, p := range f.PipelineConfig.ProcessorConfigs {
+		processor, err := models.GetProcessorByType(p.Typ, p.Config)
+		if err != nil {
+			ginx.Bomb(http.StatusBadRequest, "get processor: %+v err: %+v", p, err)
+		}
+		event, _, err = processor.Process(rt.Ctx, event)
+		if err != nil {
+			ginx.Bomb(http.StatusBadRequest, "processor: %+v err: %+v", p, err)
+		}
+
+		if event == nil {
+			ginx.NewRender(c).Data(map[string]interface{}{
+				"event":  event,
+				"result": "event is dropped",
+			}, nil)
+			return
+		}
+	}
+
+	ginx.NewRender(c).Data(event, nil)
+}
+
+// 测试事件处理器
+func (rt *Router) tryRunEventProcessor(c *gin.Context) {
+	var f struct {
+		EventId         int64                  `json:"event_id"`
+		ProcessorConfig models.ProcessorConfig `json:"processor_config"`
+	}
+	ginx.BindJSON(c, &f)
+
+	hisEvent, err := models.AlertHisEventGetById(rt.Ctx, f.EventId)
+	if err != nil || hisEvent == nil {
+		ginx.Bomb(http.StatusBadRequest, "event not found")
+	}
+	event := hisEvent.ToCur()
+
+	processor, err := models.GetProcessorByType(f.ProcessorConfig.Typ, f.ProcessorConfig.Config)
+	if err != nil {
+		ginx.Bomb(http.StatusBadRequest, "get processor err: %+v", err)
+	}
+	event, res, err := processor.Process(rt.Ctx, event)
+	if err != nil {
+		ginx.Bomb(http.StatusBadRequest, "processor err: %+v", err)
+	}
+
+	ginx.NewRender(c).Data(map[string]interface{}{
+		"event":  event,
+		"result": res,
+	}, nil)
+}
+
+func (rt *Router) tryRunEventProcessorByNotifyRule(c *gin.Context) {
+	var f struct {
+		EventId         int64                   `json:"event_id"`
+		PipelineConfigs []models.PipelineConfig `json:"pipeline_configs"`
+	}
+	ginx.BindJSON(c, &f)
+
+	hisEvent, err := models.AlertHisEventGetById(rt.Ctx, f.EventId)
+	if err != nil || hisEvent == nil {
+		ginx.Bomb(http.StatusBadRequest, "event not found")
+	}
+	event := hisEvent.ToCur()
+
+	pids := make([]int64, 0)
+	for _, pc := range f.PipelineConfigs {
+		if pc.Enable {
+			pids = append(pids, pc.PipelineId)
+		}
+	}
+
+	pipelines, err := models.GetEventPipelinesByIds(rt.Ctx, pids)
+	if err != nil {
+		ginx.Bomb(http.StatusBadRequest, "processors not found")
+	}
+
+	for _, pl := range pipelines {
+		for _, p := range pl.ProcessorConfigs {
+			processor, err := models.GetProcessorByType(p.Typ, p.Config)
+			if err != nil {
+				ginx.Bomb(http.StatusBadRequest, "get processor: %+v err: %+v", p, err)
+			}
+
+			event, _, err := processor.Process(rt.Ctx, event)
+			if err != nil {
+				ginx.Bomb(http.StatusBadRequest, "processor: %+v err: %+v", p, err)
+			}
+			if event == nil {
+				ginx.NewRender(c).Data(map[string]interface{}{
+					"event":  event,
+					"result": "event is dropped",
+				}, nil)
+				return
+			}
+		}
+	}
+
+	ginx.NewRender(c).Data(event, nil)
+}
+
+func (rt *Router) eventPipelinesListByService(c *gin.Context) {
+	pipelines, err := models.ListEventPipelines(rt.Ctx)
+	ginx.NewRender(c).Data(pipelines, err)
+}

center/router/router_funcs.go

@@ -36,6 +36,14 @@ func (rt *Router) statistic(c *gin.Context) {
 		model = models.User{}
 	case "user_group":
 		model = models.UserGroup{}
+	case "notify_rule":
+		model = models.NotifyRule{}
+	case "notify_channel":
+		model = models.NotifyChannel{}
+	case "event_pipeline":
+		statistics, err = models.EventPipelineStatistics(rt.Ctx)
+		ginx.NewRender(c).Data(statistics, err)
+		return
 	case "datasource":
 		// datasource update_at is different from others
 		statistics, err = models.DatasourceStatistics(rt.Ctx)
@@ -49,6 +57,10 @@ func (rt *Router) statistic(c *gin.Context) {
 		statistics, err = models.ConfigCvalStatistics(rt.Ctx)
 		ginx.NewRender(c).Data(statistics, err)
 		return
+	case "message_template":
+		statistics, err = models.MessageTemplateStatistics(rt.Ctx)
+		ginx.NewRender(c).Data(statistics, err)
+		return
 	default:
 		ginx.Bomb(http.StatusBadRequest, "invalid name")
 	}
@@ -161,3 +173,38 @@ func Username(c *gin.Context) string {
 	}
 	return username
 }
+
+func HasPermission(ctx *ctx.Context, c *gin.Context, sourceType, sourceId string, isAnonymousAccess bool) bool {
+	if sourceType == "event" && isAnonymousAccess {
+		return true
+	}
+
+	// 尝试从请求中获取 __token 参数
+	token := ginx.QueryStr(c, "__token", "")
+
+	// 如果有 __token 参数，验证其合法性
+	if token != "" {
+		return ValidateSourceToken(ctx, sourceType, sourceId, token)
+	}
+
+	return false
+}
+
+func ValidateSourceToken(ctx *ctx.Context, sourceType, sourceId, token string) bool {
+	if token == "" {
+		return false
+	}
+
+	// 根据源类型、源ID和令牌获取源令牌记录
+	sourceToken, err := models.GetSourceTokenBySource(ctx, sourceType, sourceId, token)
+	if err != nil {
+		return false
+	}
+
+	// 检查令牌是否过期
+	if sourceToken.IsExpired() {
+		return false
+	}
+
+	return true
+}

center/router/router_login.go

@@ -152,6 +152,13 @@ func (rt *Router) refreshPost(c *gin.Context) {
 			return
 		}
 
+		// 看这个 token 是否还存在 redis 中
+		val, err := rt.fetchAuth(c.Request.Context(), refreshUuid)
+		if err != nil || val == "" {
+			ginx.NewRender(c, http.StatusUnauthorized).Message("refresh token expired")
+			return
+		}
+
 		userIdentity, ok := claims["user_identity"].(string)
 		if !ok {
 			// Theoretically impossible

center/router/router_message_template.go

@@ -0,0 +1,219 @@
+package router
+
+import (
+	"bytes"
+	"fmt"
+	"html/template"
+	"net/http"
+	"strings"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/slice"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
+	"github.com/ccfos/nightingale/v6/pkg/tplx"
+
+	"github.com/gin-gonic/gin"
+	"github.com/google/uuid"
+	"github.com/toolkits/pkg/ginx"
+)
+
+func (rt *Router) messageTemplatesAdd(c *gin.Context) {
+	var lst []*models.MessageTemplate
+	ginx.BindJSON(c, &lst)
+	if len(lst) == 0 {
+		ginx.Bomb(http.StatusBadRequest, "input json is empty")
+	}
+
+	me := c.MustGet("user").(*models.User)
+	isAdmin := me.IsAdmin()
+	idents := make([]string, 0, len(lst))
+	gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+	ginx.Dangerous(err)
+	now := time.Now().Unix()
+	for _, tpl := range lst {
+		// 生成一个唯一的标识符，以后也不允许修改，前端不需要传这个参数
+		tpl.Ident = uuid.New().String()
+
+		ginx.Dangerous(tpl.Verify())
+		if !isAdmin && !slice.HaveIntersection(gids, tpl.UserGroupIds) {
+			ginx.Bomb(http.StatusForbidden, "forbidden")
+		}
+		idents = append(idents, tpl.Ident)
+
+		tpl.CreateBy = me.Username
+		tpl.CreateAt = now
+		tpl.UpdateBy = me.Username
+		tpl.UpdateAt = now
+	}
+
+	lstWithSameId, err := models.MessageTemplatesGet(rt.Ctx, "ident IN ?", idents)
+	ginx.Dangerous(err)
+	if len(lstWithSameId) > 0 {
+		ginx.Bomb(http.StatusBadRequest, "ident already exists")
+	}
+
+	ids := make([]int64, 0, len(lst))
+	for _, tpl := range lst {
+		err := models.Insert(rt.Ctx, tpl)
+		ginx.Dangerous(err)
+
+		ids = append(ids, tpl.ID)
+	}
+	ginx.NewRender(c).Data(ids, nil)
+}
+
+func (rt *Router) messageTemplatesDel(c *gin.Context) {
+	var f idsForm
+	ginx.BindJSON(c, &f)
+	f.Verify()
+
+	lst, err := models.MessageTemplatesGet(rt.Ctx, "id in (?)", f.Ids)
+	ginx.Dangerous(err)
+	notifyRuleIds, err := models.UsedByNotifyRule(rt.Ctx, models.MsgTplList(lst))
+	ginx.Dangerous(err)
+	if len(notifyRuleIds) > 0 {
+		ginx.NewRender(c).Message(fmt.Errorf("used by notify rule: %v", notifyRuleIds))
+		return
+	}
+	if me := c.MustGet("user").(*models.User); !me.IsAdmin() {
+		gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+		ginx.Dangerous(err)
+		for _, t := range lst {
+			if !slice.HaveIntersection(gids, t.UserGroupIds) {
+				ginx.Bomb(http.StatusForbidden, "forbidden")
+			}
+		}
+	}
+
+	ginx.NewRender(c).Message(models.DB(rt.Ctx).Delete(
+		&models.MessageTemplate{}, "id in (?)", f.Ids).Error)
+}
+
+func (rt *Router) messageTemplatePut(c *gin.Context) {
+	var f models.MessageTemplate
+	ginx.BindJSON(c, &f)
+
+	mt, err := models.MessageTemplateGet(rt.Ctx, "id <> ? and ident = ?", ginx.UrlParamInt64(c, "id"), f.Ident)
+	ginx.Dangerous(err)
+	if mt != nil {
+		ginx.Bomb(http.StatusBadRequest, "message template ident already exists")
+	}
+
+	mt, err = models.MessageTemplateGet(rt.Ctx, "id = ?", ginx.UrlParamInt64(c, "id"))
+	ginx.Dangerous(err)
+	if mt == nil {
+		ginx.Bomb(http.StatusNotFound, "message template not found")
+	}
+
+	me := c.MustGet("user").(*models.User)
+	if !me.IsAdmin() {
+		gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+		ginx.Dangerous(err)
+		if !slice.HaveIntersection(gids, mt.UserGroupIds) {
+			ginx.Bomb(http.StatusForbidden, "forbidden")
+		}
+	}
+
+	f.UpdateBy = me.Username
+	ginx.NewRender(c).Message(mt.Update(rt.Ctx, f))
+}
+
+func (rt *Router) messageTemplateGet(c *gin.Context) {
+	me := c.MustGet("user").(*models.User)
+	gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+	ginx.Dangerous(err)
+
+	tid := ginx.UrlParamInt64(c, "id")
+	mt, err := models.MessageTemplateGet(rt.Ctx, "id = ?", tid)
+	ginx.Dangerous(err)
+	if mt == nil {
+		ginx.Bomb(http.StatusNotFound, "message template not found")
+	}
+	if mt.Private == 1 && !slice.HaveIntersection(gids, mt.UserGroupIds) {
+		ginx.Bomb(http.StatusForbidden, "forbidden")
+	}
+
+	ginx.NewRender(c).Data(mt, nil)
+}
+
+func (rt *Router) messageTemplatesGet(c *gin.Context) {
+	var notifyChannelIdents []string
+	if tmp := ginx.QueryStr(c, "notify_channel_idents", ""); tmp != "" {
+		notifyChannelIdents = strings.Split(tmp, ",")
+	}
+	notifyChannelIds := strx.IdsInt64ForAPI(ginx.QueryStr(c, "notify_channel_ids", ""))
+	if len(notifyChannelIds) > 0 {
+		ginx.Dangerous(models.DB(rt.Ctx).Model(models.NotifyChannelConfig{}).
+			Where("id in (?)", notifyChannelIds).Pluck("ident", &notifyChannelIdents).Error)
+	}
+
+	me := c.MustGet("user").(*models.User)
+	gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+	ginx.Dangerous(err)
+
+	lst, err := models.MessageTemplatesGetBy(rt.Ctx, notifyChannelIdents)
+	ginx.Dangerous(err)
+
+	if me.IsAdmin() {
+		ginx.NewRender(c).Data(lst, nil)
+		return
+	}
+
+	res := make([]*models.MessageTemplate, 0)
+	for _, t := range lst {
+		if slice.HaveIntersection[int64](gids, t.UserGroupIds) || t.Private == 0 {
+			res = append(res, t)
+		}
+	}
+	ginx.NewRender(c).Data(res, nil)
+}
+
+type evtMsgReq struct {
+	EventIds []int64 `json:"event_ids"`
+	Tpl      struct {
+		Content map[string]string `json:"content"`
+	} `json:"tpl"`
+}
+
+func (rt *Router) eventsMessage(c *gin.Context) {
+	var req evtMsgReq
+	ginx.BindJSON(c, &req)
+
+	hisEvents, err := models.AlertHisEventGetByIds(rt.Ctx, req.EventIds)
+	ginx.Dangerous(err)
+
+	if len(hisEvents) == 0 {
+		ginx.Bomb(http.StatusBadRequest, "event not found")
+	}
+
+	ginx.Dangerous(err)
+	events := make([]*models.AlertCurEvent, len(hisEvents))
+	for i, he := range hisEvents {
+		events[i] = he.ToCur()
+	}
+
+	var defs = []string{
+		"{{$events := .}}",
+		"{{$event := index . 0}}",
+	}
+	ret := make(map[string]string, len(req.Tpl.Content))
+	for k, v := range req.Tpl.Content {
+		text := strings.Join(append(defs, v), "")
+		tpl, err := template.New(k).Funcs(tplx.TemplateFuncMap).Parse(text)
+		if err != nil {
+			ret[k] = err.Error()
+			continue
+		}
+
+		var buf bytes.Buffer
+		err = tpl.Execute(&buf, events)
+		if err != nil {
+			ret[k] = err.Error()
+			continue
+		}
+
+		ret[k] = buf.String()
+	}
+	ginx.NewRender(c).Data(ret, nil)
+}

center/router/router_mute.go

@@ -1,16 +1,18 @@
 package router
 
 import (
+	"math"
 	"net/http"
 	"strings"
 	"time"
 
 	"github.com/ccfos/nightingale/v6/alert/common"
+	"github.com/ccfos/nightingale/v6/alert/mute"
 	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 
 	"github.com/gin-gonic/gin"
 	"github.com/toolkits/pkg/ginx"
-	"github.com/toolkits/pkg/str"
 )
 
 // Return all, front-end search and paging
@@ -22,7 +24,7 @@ func (rt *Router) alertMuteGetsByBG(c *gin.Context) {
 }
 
 func (rt *Router) alertMuteGetsByGids(c *gin.Context) {
-	gids := str.IdsInt64(ginx.QueryStr(c, "gids", ""), ",")
+	gids := strx.IdsInt64ForAPI(ginx.QueryStr(c, "gids", ""), ",")
 	if len(gids) > 0 {
 		for _, gid := range gids {
 			rt.bgroCheck(c, gid)
@@ -63,10 +65,45 @@ func (rt *Router) alertMuteAdd(c *gin.Context) {
 
 	username := c.MustGet("username").(string)
 	f.CreateBy = username
+	f.UpdateBy = username
 	f.GroupId = ginx.UrlParamInt64(c, "id")
 	ginx.NewRender(c).Message(f.Add(rt.Ctx))
 }
 
+type MuteTestForm struct {
+	EventId   int64            `json:"event_id" binding:"required"`
+	AlertMute models.AlertMute `json:"mute_config" binding:"required"`
+}
+
+func (rt *Router) alertMuteTryRun(c *gin.Context) {
+
+	var f MuteTestForm
+	ginx.BindJSON(c, &f)
+
+	hisEvent, err := models.AlertHisEventGetById(rt.Ctx, f.EventId)
+	ginx.Dangerous(err)
+
+	if hisEvent == nil {
+		ginx.Bomb(http.StatusNotFound, "event not found")
+	}
+
+	curEvent := *hisEvent.ToCur()
+	curEvent.SetTagsMap()
+
+	// 绕过时间范围检查：设置时间范围为全量（0 到 int64 最大值），仅验证其他匹配条件（如标签、策略类型等）
+	f.AlertMute.MuteTimeType = models.TimeRange
+	f.AlertMute.Btime = 0             // 最小可能值（如 Unix 时间戳起点）
+	f.AlertMute.Etime = math.MaxInt64 // 最大可能值（int64 上限）
+
+	if !mute.MatchMute(&curEvent, &f.AlertMute) {
+		ginx.NewRender(c).Data("not match", nil)
+		return
+	}
+
+	ginx.NewRender(c).Data("mute test match", nil)
+
+}
+
 // Preview events (alert_cur_event) that match the mute strategy based on the following criteria:
 // business group ID (group_id, group_id), product (prod, rule_prod),
 // alert event severity (severities, severity), and event tags (tags, tags).

center/router/router_mw.go

@@ -9,6 +9,7 @@ import (
 	"strings"
 	"time"
 
+	"github.com/ccfos/nightingale/v6/center/cstats"
 	"github.com/ccfos/nightingale/v6/models"
 
 	"github.com/gin-gonic/gin"
@@ -335,6 +336,12 @@ func (rt *Router) extractTokenMetadata(r *http.Request) (*AccessDetails, error)
 			return nil, errors.New("failed to parse access_uuid from jwt")
 		}
 
+		// accessUuid 在 redis 里存在才放行
+		val, err := rt.fetchAuth(r.Context(), accessUuid)
+		if err != nil || val == "" {
+			return nil, errors.New("unauthorized")
+		}
+
 		return &AccessDetails{
 			AccessUuid:   accessUuid,
 			UserIdentity: claims["user_identity"].(string),
@@ -355,29 +362,72 @@ func (rt *Router) extractToken(r *http.Request) string {
 }
 
 func (rt *Router) createAuth(ctx context.Context, userIdentity string, td *TokenDetails) error {
+	username := strings.Split(userIdentity, "-")[1]
+
+	// 如果只能有一个账号登录，那么就删除之前的 token
+	if rt.HTTP.JWTAuth.SingleLogin {
+		delKeys, err := rt.Redis.SMembers(ctx, rt.wrapJwtKey(username)).Result()
+		if err != nil {
+			return err
+		}
+
+		if len(delKeys) > 0 {
+			errDel := rt.Redis.Del(ctx, delKeys...).Err()
+			if errDel != nil {
+				return errDel
+			}
+		}
+
+		if errDel := rt.Redis.Del(ctx, rt.wrapJwtKey(username)).Err(); errDel != nil {
+			return errDel
+		}
+	}
+
 	at := time.Unix(td.AtExpires, 0)
 	rte := time.Unix(td.RtExpires, 0)
 	now := time.Now()
 
-	errAccess := rt.Redis.Set(ctx, rt.wrapJwtKey(td.AccessUuid), userIdentity, at.Sub(now)).Err()
-	if errAccess != nil {
-		return errAccess
+	if err := rt.Redis.Set(ctx, rt.wrapJwtKey(td.AccessUuid), userIdentity, at.Sub(now)).Err(); err != nil {
+		cstats.RedisOperationLatency.WithLabelValues("set_token", "fail").Observe(time.Since(now).Seconds())
+		return err
 	}
 
-	errRefresh := rt.Redis.Set(ctx, rt.wrapJwtKey(td.RefreshUuid), userIdentity, rte.Sub(now)).Err()
-	if errRefresh != nil {
-		return errRefresh
+	if err := rt.Redis.Set(ctx, rt.wrapJwtKey(td.RefreshUuid), userIdentity, rte.Sub(now)).Err(); err != nil {
+		cstats.RedisOperationLatency.WithLabelValues("set_token", "fail").Observe(time.Since(now).Seconds())
+		return err
+	}
+
+	cstats.RedisOperationLatency.WithLabelValues("set_token", "success").Observe(time.Since(now).Seconds())
+
+	if rt.HTTP.JWTAuth.SingleLogin {
+		if err := rt.Redis.SAdd(ctx, rt.wrapJwtKey(username), rt.wrapJwtKey(td.AccessUuid), rt.wrapJwtKey(td.RefreshUuid)).Err(); err != nil {
+			return err
+		}
 	}
 
 	return nil
 }
 
 func (rt *Router) fetchAuth(ctx context.Context, givenUuid string) (string, error) {
-	return rt.Redis.Get(ctx, rt.wrapJwtKey(givenUuid)).Result()
+	now := time.Now()
+	ret, err := rt.Redis.Get(ctx, rt.wrapJwtKey(givenUuid)).Result()
+	if err != nil {
+		cstats.RedisOperationLatency.WithLabelValues("get_token", "fail").Observe(time.Since(now).Seconds())
+	} else {
+		cstats.RedisOperationLatency.WithLabelValues("get_token", "success").Observe(time.Since(now).Seconds())
+	}
+
+	return ret, err
 }
 
 func (rt *Router) deleteAuth(ctx context.Context, givenUuid string) error {
-	return rt.Redis.Del(ctx, rt.wrapJwtKey(givenUuid)).Err()
+	err := rt.Redis.Del(ctx, rt.wrapJwtKey(givenUuid)).Err()
+	if err != nil {
+		cstats.RedisOperationLatency.WithLabelValues("del_token", "fail").Observe(time.Since(time.Now()).Seconds())
+	} else {
+		cstats.RedisOperationLatency.WithLabelValues("del_token", "success").Observe(time.Since(time.Now()).Seconds())
+	}
+	return err
 }
 
 func (rt *Router) deleteTokens(ctx context.Context, authD *AccessDetails) error {

center/router/router_notification_record.go

@@ -18,20 +18,17 @@ type NotificationResponse struct {
 }
 
 type SubRule struct {
-	SubID    int64               `json:"sub_id"`
-	Notifies map[string][]Record `json:"notifies"`
-}
-
-type Notify struct {
-	Channel string   `json:"channel"`
-	Records []Record `json:"records"`
+	SubID        int64               `json:"sub_id"`
+	NotifyRuleId int64               `json:"notify_rule_id"`
+	Notifies     map[string][]Record `json:"notifies"`
 }
 
 type Record struct {
-	Target   string `json:"target"`
-	Username string `json:"username"`
-	Status   int    `json:"status"`
-	Detail   string `json:"detail"`
+	NotifyRuleId int64  `json:"notify_rule_id"`
+	Target       string `json:"target"`
+	Username     string `json:"username"`
+	Status       int    `json:"status"`
+	Detail       string `json:"detail"`
 }
 
 // notificationRecordAdd
@@ -40,7 +37,7 @@ func (rt *Router) notificationRecordAdd(c *gin.Context) {
 	ginx.BindJSON(c, &req)
 	err := sender.PushNotifyRecords(req)
 	ginx.Dangerous(err, 429)
-	
+
 	ginx.NewRender(c).Data(nil, err)
 }
 
@@ -113,9 +110,10 @@ func buildNotificationResponse(ctx *ctx.Context, nl []*models.NotificaitonRecord
 			n.Target = replaceLastEightChars(n.Target)
 		}
 		record := Record{
-			Target: n.Target,
-			Status: n.Status,
-			Detail: n.Details,
+			Target:       n.Target,
+			Status:       n.Status,
+			Detail:       n.Details,
+			NotifyRuleId: n.NotifyRuleID,
 		}
 
 		record.Username = strings.Join(usernames, ",")
@@ -125,7 +123,8 @@ func buildNotificationResponse(ctx *ctx.Context, nl []*models.NotificaitonRecord
 			subRule, ok := subRuleMap[n.SubId]
 			if !ok {
 				newSubRule := &SubRule{
-					SubID: n.SubId,
+					NotifyRuleId: n.NotifyRuleID,
+					SubID:        n.SubId,
 				}
 				newSubRule.Notifies = make(map[string][]Record)
 				newSubRule.Notifies[n.Channel] = []Record{record}

center/router/router_notify_channel.go

@@ -0,0 +1,268 @@
+package router
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"sort"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/gin-gonic/gin"
+	"github.com/toolkits/pkg/ginx"
+)
+
+func (rt *Router) notifyChannelsAdd(c *gin.Context) {
+	me := c.MustGet("user").(*models.User)
+
+	var lst []*models.NotifyChannelConfig
+	ginx.BindJSON(c, &lst)
+	if len(lst) == 0 {
+		ginx.Bomb(http.StatusBadRequest, "input json is empty")
+	}
+
+	names := make([]string, 0, len(lst))
+	for i := range lst {
+		ginx.Dangerous(lst[i].Verify())
+		names = append(names, lst[i].Name)
+
+		lst[i].CreateBy = me.Username
+		lst[i].CreateAt = time.Now().Unix()
+		lst[i].UpdateBy = me.Username
+		lst[i].UpdateAt = time.Now().Unix()
+	}
+
+	lstWithSameName, err := models.NotifyChannelsGet(rt.Ctx, "name IN ?", names)
+	ginx.Dangerous(err)
+	if len(lstWithSameName) > 0 {
+		ginx.Bomb(http.StatusBadRequest, "name already exists")
+	}
+
+	ids := make([]int64, 0, len(lst))
+	for _, item := range lst {
+		err := models.Insert(rt.Ctx, item)
+		ginx.Dangerous(err)
+
+		ids = append(ids, item.ID)
+	}
+	ginx.NewRender(c).Data(ids, nil)
+}
+
+func (rt *Router) notifyChannelsDel(c *gin.Context) {
+	var f idsForm
+	ginx.BindJSON(c, &f)
+	f.Verify()
+
+	lst, err := models.NotifyChannelsGet(rt.Ctx, "id in (?)", f.Ids)
+	ginx.Dangerous(err)
+	notifyRuleIds, err := models.UsedByNotifyRule(rt.Ctx, models.NotiChList(lst))
+	ginx.Dangerous(err)
+	if len(notifyRuleIds) > 0 {
+		ginx.NewRender(c).Message(fmt.Errorf("used by notify rule: %v", notifyRuleIds))
+		return
+	}
+
+	ginx.NewRender(c).Message(models.DB(rt.Ctx).
+		Delete(&models.NotifyChannelConfig{}, "id in (?)", f.Ids).Error)
+}
+
+func (rt *Router) notifyChannelPut(c *gin.Context) {
+	me := c.MustGet("user").(*models.User)
+
+	var f models.NotifyChannelConfig
+	ginx.BindJSON(c, &f)
+
+	lstWithSameName, err := models.NotifyChannelsGet(rt.Ctx, "name = ? and id <> ?", f.Name, f.ID)
+	ginx.Dangerous(err)
+	if len(lstWithSameName) > 0 {
+		ginx.Bomb(http.StatusBadRequest, "name already exists")
+	}
+
+	nc, err := models.NotifyChannelGet(rt.Ctx, "id = ?", ginx.UrlParamInt64(c, "id"))
+	ginx.Dangerous(err)
+	if nc == nil {
+		ginx.Bomb(http.StatusNotFound, "notify channel not found")
+	}
+
+	f.UpdateBy = me.Username
+	ginx.NewRender(c).Message(nc.Update(rt.Ctx, f))
+}
+
+func (rt *Router) notifyChannelGet(c *gin.Context) {
+	cid := ginx.UrlParamInt64(c, "id")
+	nc, err := models.NotifyChannelGet(rt.Ctx, "id = ?", cid)
+	ginx.Dangerous(err)
+	if nc == nil {
+		ginx.Bomb(http.StatusNotFound, "notify channel not found")
+	}
+
+	ginx.NewRender(c).Data(nc, nil)
+}
+
+func (rt *Router) notifyChannelGetBy(c *gin.Context) {
+	ident := ginx.QueryStr(c, "ident")
+	nc, err := models.NotifyChannelGet(rt.Ctx, "ident = ?", ident)
+	ginx.Dangerous(err)
+	if nc == nil {
+		ginx.Bomb(http.StatusNotFound, "notify channel not found")
+	}
+
+	nc.ParamConfig = &models.NotifyParamConfig{}
+	nc.RequestConfig = &models.RequestConfig{}
+
+	ginx.NewRender(c).Data(nc, nil)
+}
+
+func (rt *Router) notifyChannelsGet(c *gin.Context) {
+	lst, err := models.NotifyChannelsGet(rt.Ctx, "", nil)
+	ginx.NewRender(c).Data(lst, err)
+}
+
+func (rt *Router) notifyChannelsGetForNormalUser(c *gin.Context) {
+	lst, err := models.NotifyChannelsGet(rt.Ctx, "")
+	ginx.Dangerous(err)
+
+	newLst := make([]*models.NotifyChannelConfig, 0, len(lst))
+	for _, c := range lst {
+		newLst = append(newLst, &models.NotifyChannelConfig{
+			ID:          c.ID,
+			Name:        c.Name,
+			Ident:       c.Ident,
+			Enable:      c.Enable,
+			RequestType: c.RequestType,
+			ParamConfig: c.ParamConfig,
+		})
+	}
+	ginx.NewRender(c).Data(newLst, nil)
+}
+
+func (rt *Router) notifyChannelIdentsGet(c *gin.Context) {
+	// 获取所有通知渠道
+	channels, err := models.NotifyChannelsGet(rt.Ctx, "", nil)
+	ginx.Dangerous(err)
+
+	// ident 去重
+	idents := make(map[string]struct{})
+	for _, channel := range channels {
+		if channel.Ident != "" {
+			idents[channel.Ident] = struct{}{}
+		}
+	}
+
+	lst := make([]string, 0, len(idents))
+	for ident := range idents {
+		lst = append(lst, ident)
+	}
+
+	sort.Strings(lst)
+
+	ginx.NewRender(c).Data(lst, nil)
+}
+
+type flushDutyChannelsResponse struct {
+	Error struct {
+		Code    string `json:"code"`
+		Message string `json:"message"`
+	} `json:"error"`
+	Data struct {
+		Items []struct {
+			ChannelID   int    `json:"channel_id"`
+			ChannelName string `json:"channel_name"`
+			Status      string `json:"status"`
+		} `json:"items"`
+		Total int `json:"total"`
+	} `json:"data"`
+}
+
+func (rt *Router) flashDutyNotifyChannelsGet(c *gin.Context) {
+	cid := ginx.UrlParamInt64(c, "id")
+	nc, err := models.NotifyChannelGet(rt.Ctx, "id = ?", cid)
+	ginx.Dangerous(err)
+	if nc == nil {
+		ginx.Bomb(http.StatusNotFound, "notify channel not found")
+	}
+
+	configs, err := models.ConfigsSelectByCkey(rt.Ctx, "flashduty_app_key")
+	if err != nil {
+		ginx.Bomb(http.StatusInternalServerError, "failed to get flashduty app key")
+	}
+
+	jsonData := []byte("{}")
+	if len(configs) > 0 {
+		me := c.MustGet("user").(*models.User)
+		jsonData = []byte(fmt.Sprintf(`{"member_name":"%s","email":"%s","phone":"%s"}`, me.Username, me.Email, me.Phone))
+	}
+
+	items, err := getFlashDutyChannels(nc.RequestConfig.FlashDutyRequestConfig.IntegrationUrl, jsonData)
+	ginx.Dangerous(err)
+
+	ginx.NewRender(c).Data(items, nil)
+}
+
+// getFlashDutyChannels 从FlashDuty API获取频道列表
+func getFlashDutyChannels(integrationUrl string, jsonData []byte) ([]struct {
+	ChannelID   int    `json:"channel_id"`
+	ChannelName string `json:"channel_name"`
+	Status      string `json:"status"`
+}, error) {
+	// 解析URL，提取baseUrl和参数
+	baseUrl, integrationKey, err := parseIntegrationUrl(integrationUrl)
+	if err != nil {
+		return nil, err
+	}
+
+	if integrationKey == "" {
+		return nil, fmt.Errorf("integration_key not found in URL")
+	}
+
+	// 构建新的API URL，保持原始路径
+	url := fmt.Sprintf("%s/channel/list-by-integration?integration_key=%s", baseUrl, integrationKey)
+
+	req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonData))
+	if err != nil {
+		return nil, err
+	}
+
+	req.Header.Set("Content-Type", "application/json")
+	httpResp, err := (&http.Client{}).Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer httpResp.Body.Close()
+
+	body, err := io.ReadAll(httpResp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	var res flushDutyChannelsResponse
+	if err := json.Unmarshal(body, &res); err != nil {
+		return nil, err
+	}
+
+	if res.Error.Message != "" {
+		return nil, fmt.Errorf(res.Error.Message)
+	}
+
+	return res.Data.Items, nil
+}
+
+// parseIntegrationUrl 从URL中提取baseUrl和参数
+func parseIntegrationUrl(urlStr string) (baseUrl string, integrationKey string, err error) {
+	// 解析URL
+	parsedUrl, err := url.Parse(urlStr)
+	if err != nil {
+		return "", "", err
+	}
+
+	host := fmt.Sprintf("%s://%s", parsedUrl.Scheme, parsedUrl.Host)
+
+	// 提取查询参数
+	queryParams := parsedUrl.Query()
+	integrationKey = queryParams.Get("integration_key")
+
+	return host, integrationKey, nil
+}

center/router/router_notify_channel_test.go

@@ -0,0 +1,17 @@
+package router
+
+import (
+	"fmt"
+	"testing"
+)
+
+func TestGetFlashDutyChannels(t *testing.T) {
+	// 构造测试数据
+	integrationUrl := "https://api.flashcat.cloud/event/push/alert/n9e?integration_key=xxx"
+	jsonData := []byte(`{}`)
+
+	// 调用被测试的函数
+	channels, err := getFlashDutyChannels(integrationUrl, jsonData)
+
+	fmt.Println(channels, err)
+}

center/router/router_notify_config.go

@@ -111,7 +111,7 @@ func (rt *Router) notifyChannelPuts(c *gin.Context) {
 }
 
 func (rt *Router) notifyContactGets(c *gin.Context) {
-	var notifyContacts []models.NotifyContact
+	notifyContacts := []models.NotifyContact{}
 	cval, err := models.ConfigsGet(rt.Ctx, models.NOTIFYCONTACT)
 	ginx.Dangerous(err)
 	if cval == "" {
@@ -120,6 +120,7 @@ func (rt *Router) notifyContactGets(c *gin.Context) {
 	}
 
 	err = json.Unmarshal([]byte(cval), &notifyContacts)
+
 	ginx.NewRender(c).Data(notifyContacts, err)
 }
 
@@ -127,20 +128,6 @@ func (rt *Router) notifyContactPuts(c *gin.Context) {
 	var notifyContacts []models.NotifyContact
 	ginx.BindJSON(c, &notifyContacts)
 
-	keys := []string{models.DingtalkKey, models.WecomKey, models.FeishuKey, models.MmKey,
-		models.TelegramKey, models.LarkKey}
-
-	m := make(map[string]struct{})
-	for _, v := range notifyContacts {
-		m[v.Ident] = struct{}{}
-	}
-
-	for _, v := range keys {
-		if _, ok := m[v]; !ok {
-			ginx.Bomb(200, "contact %s ident can not modify", v)
-		}
-	}
-
 	data, err := json.Marshal(notifyContacts)
 	ginx.Dangerous(err)
 	username := c.MustGet("username").(string)
@@ -226,3 +213,14 @@ func (rt *Router) attemptSendEmail(c *gin.Context) {
 	ginx.NewRender(c).Message(sender.SendEmail("Email test", "email content", []string{f.Email}, smtp))
 
 }
+
+func (rt *Router) notifyChannelConfigGets(c *gin.Context) {
+
+	id := ginx.QueryInt64(c, "id", 0)
+	name := ginx.QueryStr(c, "name", "")
+	ident := ginx.QueryStr(c, "ident", "")
+	eabled := ginx.QueryInt(c, "eabled", -1)
+
+	notifyChannels, err := models.NotifyChannelGets(rt.Ctx, id, name, ident, eabled)
+	ginx.NewRender(c).Data(notifyChannels, err)
+}

center/router/router_notify_rule.go

@@ -0,0 +1,324 @@
+package router
+
+import (
+	"fmt"
+	"net/http"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/alert/dispatch"
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/slice"
+
+	"github.com/gin-gonic/gin"
+	"github.com/pkg/errors"
+	"github.com/toolkits/pkg/ginx"
+	"github.com/toolkits/pkg/logger"
+)
+
+func (rt *Router) notifyRulesAdd(c *gin.Context) {
+	var lst []*models.NotifyRule
+	ginx.BindJSON(c, &lst)
+	if len(lst) == 0 {
+		ginx.Bomb(http.StatusBadRequest, "input json is empty")
+	}
+
+	me := c.MustGet("user").(*models.User)
+	isAdmin := me.IsAdmin()
+	gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+	ginx.Dangerous(err)
+
+	now := time.Now().Unix()
+	for _, nr := range lst {
+		ginx.Dangerous(nr.Verify())
+		if !isAdmin && !slice.HaveIntersection(gids, nr.UserGroupIds) {
+			ginx.Bomb(http.StatusForbidden, "forbidden")
+		}
+
+		nr.CreateBy = me.Username
+		nr.CreateAt = now
+		nr.UpdateBy = me.Username
+		nr.UpdateAt = now
+
+		err := models.Insert(rt.Ctx, nr)
+		ginx.Dangerous(err)
+	}
+	ginx.NewRender(c).Data(lst, nil)
+}
+
+func (rt *Router) notifyRulesDel(c *gin.Context) {
+	var f idsForm
+	ginx.BindJSON(c, &f)
+	f.Verify()
+
+	if me := c.MustGet("user").(*models.User); !me.IsAdmin() {
+		lst, err := models.NotifyRulesGet(rt.Ctx, "id in (?)", f.Ids)
+		ginx.Dangerous(err)
+		gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+		ginx.Dangerous(err)
+		for _, t := range lst {
+			if !slice.HaveIntersection(gids, t.UserGroupIds) {
+				ginx.Bomb(http.StatusForbidden, "forbidden")
+			}
+		}
+	}
+
+	ginx.NewRender(c).Message(models.DB(rt.Ctx).
+		Delete(&models.NotifyRule{}, "id in (?)", f.Ids).Error)
+}
+
+func (rt *Router) notifyRulePut(c *gin.Context) {
+	var f models.NotifyRule
+	ginx.BindJSON(c, &f)
+
+	nr, err := models.NotifyRuleGet(rt.Ctx, "id = ?", ginx.UrlParamInt64(c, "id"))
+	ginx.Dangerous(err)
+	if nr == nil {
+		ginx.Bomb(http.StatusNotFound, "notify rule not found")
+	}
+
+	me := c.MustGet("user").(*models.User)
+	gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+	ginx.Dangerous(err)
+	if !slice.HaveIntersection(gids, nr.UserGroupIds) && !me.IsAdmin() {
+		ginx.Bomb(http.StatusForbidden, "forbidden")
+	}
+
+	f.UpdateBy = me.Username
+	ginx.NewRender(c).Message(nr.Update(rt.Ctx, f))
+}
+
+func (rt *Router) notifyRuleGet(c *gin.Context) {
+	me := c.MustGet("user").(*models.User)
+	gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+	ginx.Dangerous(err)
+
+	tid := ginx.UrlParamInt64(c, "id")
+	nr, err := models.NotifyRuleGet(rt.Ctx, "id = ?", tid)
+	ginx.Dangerous(err)
+	if nr == nil {
+		ginx.Bomb(http.StatusNotFound, "notify rule not found")
+	}
+
+	if !slice.HaveIntersection(gids, nr.UserGroupIds) && !me.IsAdmin() {
+		ginx.Bomb(http.StatusForbidden, "forbidden")
+	}
+
+	ginx.NewRender(c).Data(nr, nil)
+}
+
+func (rt *Router) notifyRulesGetByService(c *gin.Context) {
+	ginx.NewRender(c).Data(models.NotifyRulesGet(rt.Ctx, "enable = ?", true))
+}
+
+func (rt *Router) notifyRulesGet(c *gin.Context) {
+	me := c.MustGet("user").(*models.User)
+	gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+	ginx.Dangerous(err)
+
+	lst, err := models.NotifyRulesGet(rt.Ctx, "", nil)
+	ginx.Dangerous(err)
+	if me.IsAdmin() {
+		ginx.NewRender(c).Data(lst, nil)
+		return
+	}
+
+	res := make([]*models.NotifyRule, 0)
+	for _, nr := range lst {
+		if slice.HaveIntersection[int64](gids, nr.UserGroupIds) {
+			res = append(res, nr)
+		}
+	}
+	ginx.NewRender(c).Data(res, nil)
+}
+
+type NotifyTestForm struct {
+	EventIDs     []int64             `json:"event_ids" binding:"required"`
+	NotifyConfig models.NotifyConfig `json:"notify_config" binding:"required"`
+}
+
+func (rt *Router) notifyTest(c *gin.Context) {
+	var f NotifyTestForm
+	ginx.BindJSON(c, &f)
+
+	hisEvents, err := models.AlertHisEventGetByIds(rt.Ctx, f.EventIDs)
+	ginx.Dangerous(err)
+
+	if len(hisEvents) == 0 {
+		ginx.Bomb(http.StatusBadRequest, "event not found")
+	}
+
+	ginx.Dangerous(err)
+	events := []*models.AlertCurEvent{}
+	for _, he := range hisEvents {
+		event := he.ToCur()
+		event.SetTagsMap()
+		if dispatch.NotifyRuleApplicable(&f.NotifyConfig, event) {
+			events = append(events, event)
+		}
+	}
+
+	if len(events) == 0 {
+		ginx.Bomb(http.StatusBadRequest, "not events applicable")
+	}
+
+	notifyChannels, err := models.NotifyChannelGets(rt.Ctx, f.NotifyConfig.ChannelID, "", "", -1)
+	ginx.Dangerous(err)
+	if len(notifyChannels) == 0 {
+		ginx.Bomb(http.StatusBadRequest, "notify channel not found")
+	}
+
+	notifyChannel := notifyChannels[0]
+
+	if !notifyChannel.Enable {
+		ginx.Bomb(http.StatusBadRequest, "notify channel not enabled, please enable it first")
+	}
+
+	tplContent := make(map[string]interface{})
+	if notifyChannel.RequestType != "flashtudy" {
+		messageTemplates, err := models.MessageTemplateGets(rt.Ctx, f.NotifyConfig.TemplateID, "", "")
+		ginx.Dangerous(err)
+		if len(messageTemplates) == 0 {
+			ginx.Bomb(http.StatusBadRequest, "message template not found")
+		}
+		tplContent = messageTemplates[0].RenderEvent(events)
+	}
+
+	var contactKey string
+	if notifyChannel.ParamConfig != nil && notifyChannel.ParamConfig.UserInfo != nil {
+		contactKey = notifyChannel.ParamConfig.UserInfo.ContactKey
+	}
+
+	sendtos, flashDutyChannelIDs, customParams := dispatch.GetNotifyConfigParams(&f.NotifyConfig, contactKey, rt.UserCache, rt.UserGroupCache)
+
+	var resp string
+	switch notifyChannel.RequestType {
+	case "flashduty":
+		client, err := models.GetHTTPClient(notifyChannel)
+		ginx.Dangerous(err)
+
+		for i := range flashDutyChannelIDs {
+			resp, err = notifyChannel.SendFlashDuty(events, flashDutyChannelIDs[i], client)
+			if err != nil {
+				break
+			}
+		}
+		logger.Infof("channel_name: %v, event:%+v, tplContent:%s, customParams:%v, respBody: %v, err: %v", notifyChannel.Name, events[0], tplContent, customParams, resp, err)
+		ginx.NewRender(c).Data(resp, err)
+	case "http":
+		client, err := models.GetHTTPClient(notifyChannel)
+		ginx.Dangerous(err)
+
+		if notifyChannel.RequestConfig == nil {
+			ginx.Bomb(http.StatusBadRequest, "request config not found")
+		}
+
+		if notifyChannel.RequestConfig.HTTPRequestConfig == nil {
+			ginx.Bomb(http.StatusBadRequest, "http request config not found")
+		}
+
+		if dispatch.NeedBatchContacts(notifyChannel.RequestConfig.HTTPRequestConfig) || len(sendtos) == 0 {
+			resp, err = notifyChannel.SendHTTP(events, tplContent, customParams, sendtos, client)
+			logger.Infof("channel_name: %v, event:%+v, sendtos:%+v, tplContent:%s, customParams:%v, respBody: %v, err: %v", notifyChannel.Name, events[0], sendtos, tplContent, customParams, resp, err)
+			if err != nil {
+				logger.Errorf("failed to send http notify: %v", err)
+			}
+			ginx.NewRender(c).Data(resp, err)
+		} else {
+			for i := range sendtos {
+				resp, err = notifyChannel.SendHTTP(events, tplContent, customParams, []string{sendtos[i]}, client)
+				logger.Infof("channel_name: %v, event:%+v,  tplContent:%s, customParams:%v, sendto:%+v, respBody: %v, err: %v", notifyChannel.Name, events[0], tplContent, customParams, sendtos[i], resp, err)
+				if err != nil {
+					logger.Errorf("failed to send http notify: %v", err)
+					ginx.NewRender(c).Message(err)
+					return
+				}
+			}
+			ginx.NewRender(c).Message(err)
+		}
+
+	case "smtp":
+		if len(sendtos) == 0 {
+			ginx.Bomb(http.StatusBadRequest, "No valid email address in the user and team")
+		}
+		err := notifyChannel.SendEmailNow(events, tplContent, sendtos)
+		ginx.NewRender(c).Message(err)
+	case "script":
+		resp, _, err := notifyChannel.SendScript(events, tplContent, customParams, sendtos)
+		logger.Infof("channel_name: %v, event:%+v, tplContent:%s, customParams:%v, respBody: %v, err: %v", notifyChannel.Name, events[0], tplContent, customParams, resp, err)
+		ginx.NewRender(c).Data(resp, err)
+	default:
+		logger.Errorf("unsupported request type: %v", notifyChannel.RequestType)
+		ginx.NewRender(c).Message(errors.New("unsupported request type"))
+	}
+}
+
+type paramList struct {
+	Name  string      `json:"name"`
+	CName string      `json:"cname"`
+	Value interface{} `json:"value"`
+}
+
+func (rt *Router) notifyRuleCustomParamsGet(c *gin.Context) {
+	notifyChannelID := ginx.QueryInt64(c, "notify_channel_id")
+
+	me := c.MustGet("user").(*models.User)
+	gids, err := models.MyGroupIds(rt.Ctx, me.Id)
+	ginx.Dangerous(err)
+
+	notifyChannel, err := models.NotifyChannelGet(rt.Ctx, "id=?", notifyChannelID)
+	ginx.Dangerous(err)
+
+	keyMap := make(map[string]string)
+	if notifyChannel == nil {
+		ginx.NewRender(c).Data([][]paramList{}, nil)
+		return
+	}
+
+	if notifyChannel.ParamConfig == nil {
+		ginx.NewRender(c).Data([][]paramList{}, nil)
+		return
+	}
+
+	for _, param := range notifyChannel.ParamConfig.Custom.Params {
+		keyMap[param.Key] = param.CName
+	}
+
+	lst, err := models.NotifyRulesGet(rt.Ctx, "", nil)
+	ginx.Dangerous(err)
+
+	res := make([][]paramList, 0)
+	filter := make(map[string]struct{})
+	for _, nr := range lst {
+		if !slice.HaveIntersection[int64](gids, nr.UserGroupIds) {
+			continue
+		}
+
+		for _, nc := range nr.NotifyConfigs {
+			if nc.ChannelID != notifyChannelID {
+				continue
+			}
+
+			list := make([]paramList, 0)
+			filterKey := ""
+			for key, value := range nc.Params {
+				// 找到在通知媒介中的自定义变量配置项，进行 cname 转换
+				cname, exsits := keyMap[key]
+				if exsits {
+					list = append(list, paramList{
+						Name:  key,
+						CName: cname,
+						Value: value,
+					})
+				}
+				filterKey += fmt.Sprintf("%s:%s,", key, value)
+			}
+			if _, ok := filter[filterKey]; ok {
+				continue
+			}
+			filter[filterKey] = struct{}{}
+			res = append(res, list)
+		}
+	}
+
+	ginx.NewRender(c).Data(res, nil)
+}

center/router/router_notify_tpl.go

@@ -45,7 +45,7 @@ func (rt *Router) notifyTplUpdateContent(c *gin.Context) {
 	ginx.Dangerous(err)
 
 	if notifyTpl.CreateBy != user.Username && !user.IsAdmin() {
-		ginx.Bomb(403, "no permission")
+		ginx.Bomb(403, "forbidden")
 	}
 
 	f.UpdateAt = time.Now().Unix()
@@ -64,7 +64,7 @@ func (rt *Router) notifyTplUpdate(c *gin.Context) {
 	ginx.Dangerous(err)
 
 	if notifyTpl.CreateBy != user.Username && !user.IsAdmin() {
-		ginx.Bomb(403, "no permission")
+		ginx.Bomb(403, "forbidden")
 	}
 
 	// get the count of the same channel and name but different id
@@ -162,10 +162,10 @@ func (rt *Router) notifyTplAdd(c *gin.Context) {
 	var f models.NotifyTpl
 	ginx.BindJSON(c, &f)
 
-        user := c.MustGet("user").(*models.User)
-        f.CreateBy = user.Username
-	
-        f.Channel = strings.TrimSpace(f.Channel)
+	user := c.MustGet("user").(*models.User)
+	f.CreateBy = user.Username
+
+	f.Channel = strings.TrimSpace(f.Channel)
 	ginx.Dangerous(templateValidate(f))
 
 	count, err := models.Count(models.DB(rt.Ctx).Model(&models.NotifyTpl{}).Where("channel = ? or name = ?", f.Channel, f.Name))
@@ -174,7 +174,7 @@ func (rt *Router) notifyTplAdd(c *gin.Context) {
 		ginx.Bomb(200, "Refuse to create duplicate channel(unique)")
 	}
 
-        f.CreateAt = time.Now().Unix()
+	f.CreateAt = time.Now().Unix()
 	ginx.NewRender(c).Message(f.Create(rt.Ctx))
 }
 
@@ -188,8 +188,18 @@ func (rt *Router) notifyTplDel(c *gin.Context) {
 	ginx.Dangerous(err)
 
 	if notifyTpl.CreateBy != user.Username && !user.IsAdmin() {
-		ginx.Bomb(403, "no permission")
+		ginx.Bomb(403, "forbidden")
 	}
 
 	ginx.NewRender(c).Message(f.NotifyTplDelete(rt.Ctx, id))
 }
+
+func (rt *Router) messageTemplateGets(c *gin.Context) {
+	id := ginx.QueryInt64(c, "id", 0)
+	name := ginx.QueryStr(c, "name", "")
+	ident := ginx.QueryStr(c, "ident", "")
+
+	tpls, err := models.MessageTemplateGets(rt.Ctx, id, name, ident)
+
+	ginx.NewRender(c).Data(tpls, err)
+}

center/router/router_query.go

@@ -3,6 +3,7 @@ package router
 import (
 	"fmt"
 	"sort"
+	"sync"
 
 	"github.com/ccfos/nightingale/v6/dscache"
 	"github.com/ccfos/nightingale/v6/models"
@@ -38,71 +39,116 @@ type LogResp struct {
 	List  []interface{} `json:"list"`
 }
 
-func (rt *Router) QueryLogBatch(c *gin.Context) {
-	var f QueryFrom
-	ginx.BindJSON(c, &f)
-
+func QueryLogBatchConcurrently(anonymousAccess bool, ctx *gin.Context, f QueryFrom) (LogResp, error) {
 	var resp LogResp
-	var errMsg string
+	var mu sync.Mutex
+	var wg sync.WaitGroup
+	var errs []error
+
 	for _, q := range f.Queries {
-		if !rt.Center.AnonymousAccess.PromQuerier && !CheckDsPerm(c, q.Did, q.DsCate, q) {
-			ginx.Bomb(200, "no permission")
+		if !anonymousAccess && !CheckDsPerm(ctx, q.Did, q.DsCate, q) {
+			return LogResp{}, fmt.Errorf("forbidden")
 		}
 
 		plug, exists := dscache.DsCache.Get(q.DsCate, q.Did)
 		if !exists {
 			logger.Warningf("cluster:%d not exists query:%+v", q.Did, q)
-			ginx.Bomb(200, "cluster not exists")
+			return LogResp{}, fmt.Errorf("cluster not exists")
 		}
 
-		data, total, err := plug.QueryLog(c.Request.Context(), q.Query)
-		if err != nil {
-			errMsg += fmt.Sprintf("query data error: %v query:%v\n ", err, q)
-			logger.Warningf("query data error: %v query:%v", err, q)
-			continue
-		}
+		wg.Add(1)
+		go func(query Query) {
+			defer wg.Done()
 
-		m := make(map[string]interface{})
-		m["ref"] = q.Ref
-		m["ds_id"] = q.Did
-		m["ds_cate"] = q.DsCate
-		m["data"] = data
-		resp.List = append(resp.List, m)
-		resp.Total += total
+			data, total, err := plug.QueryLog(ctx.Request.Context(), query.Query)
+			mu.Lock()
+			defer mu.Unlock()
+			if err != nil {
+				errMsg := fmt.Sprintf("query data error: %v query:%v\n ", err, query)
+				logger.Warningf(errMsg)
+				errs = append(errs, err)
+				return
+			}
+
+			m := make(map[string]interface{})
+			m["ref"] = query.Ref
+			m["ds_id"] = query.Did
+			m["ds_cate"] = query.DsCate
+			m["data"] = data
+
+			resp.List = append(resp.List, m)
+			resp.Total += total
+		}(q)
 	}
 
-	if errMsg != "" || len(resp.List) == 0 {
-		ginx.Bomb(200, errMsg)
+	wg.Wait()
+
+	if len(errs) > 0 {
+		return LogResp{}, errs[0]
+	}
+
+	if len(resp.List) == 0 {
+		return LogResp{}, fmt.Errorf("no data")
+	}
+
+	return resp, nil
+}
+
+func (rt *Router) QueryLogBatch(c *gin.Context) {
+	var f QueryFrom
+	ginx.BindJSON(c, &f)
+
+	resp, err := QueryLogBatchConcurrently(rt.Center.AnonymousAccess.PromQuerier, c, f)
+	if err != nil {
+		ginx.Bomb(200, "err:%v", err)
 	}
 
 	ginx.NewRender(c).Data(resp, nil)
 }
 
-func (rt *Router) QueryData(c *gin.Context) {
-	var f models.QueryParam
-	ginx.BindJSON(c, &f)
-
+func QueryDataConcurrently(anonymousAccess bool, ctx *gin.Context, f models.QueryParam) ([]models.DataResp, error) {
 	var resp []models.DataResp
-	var err error
+	var mu sync.Mutex
+	var wg sync.WaitGroup
+	var errs []error
+
 	for _, q := range f.Querys {
-		if !rt.Center.AnonymousAccess.PromQuerier && !CheckDsPerm(c, f.DatasourceId, f.Cate, q) {
-			ginx.Bomb(403, "no permission")
+		if !anonymousAccess && !CheckDsPerm(ctx, f.DatasourceId, f.Cate, q) {
+			return nil, fmt.Errorf("forbidden")
 		}
 
 		plug, exists := dscache.DsCache.Get(f.Cate, f.DatasourceId)
 		if !exists {
 			logger.Warningf("cluster:%d not exists", f.DatasourceId)
-			ginx.Bomb(200, "cluster not exists")
+			return nil, fmt.Errorf("cluster not exists")
 		}
-		var datas []models.DataResp
-		datas, err = plug.QueryData(c.Request.Context(), q)
-		if err != nil {
-			logger.Warningf("query data error: req:%+v err:%v", q, err)
-			ginx.Bomb(200, "err:%v", err)
-		}
-		logger.Debugf("query data: req:%+v resp:%+v", q, datas)
-		resp = append(resp, datas...)
+
+		wg.Add(1)
+		go func(query interface{}) {
+			defer wg.Done()
+
+			datas, err := plug.QueryData(ctx.Request.Context(), query)
+			if err != nil {
+				logger.Warningf("query data error: req:%+v err:%v", query, err)
+				mu.Lock()
+				errs = append(errs, err)
+				mu.Unlock()
+				return
+			}
+
+			logger.Debugf("query data: req:%+v resp:%+v", query, datas)
+			mu.Lock()
+			resp = append(resp, datas...)
+			mu.Unlock()
+		}(q)
 	}
+
+	wg.Wait()
+
+	if len(errs) > 0 {
+		return nil, errs[0]
+	}
+
 	// 面向API的统一处理
 	// 按照 .Metric 排序
 	// 确保仪表盘中相同图例的曲线颜色相同
@@ -115,41 +161,80 @@ func (rt *Router) QueryData(c *gin.Context) {
 		})
 	}
 
-	ginx.NewRender(c).Data(resp, err)
+	return resp, nil
+}
+
+func (rt *Router) QueryData(c *gin.Context) {
+	var f models.QueryParam
+	ginx.BindJSON(c, &f)
+
+	resp, err := QueryDataConcurrently(rt.Center.AnonymousAccess.PromQuerier, c, f)
+	if err != nil {
+		ginx.Bomb(200, "err:%v", err)
+	}
+
+	ginx.NewRender(c).Data(resp, nil)
+}
+
+// QueryLogConcurrently 并发查询日志
+func QueryLogConcurrently(anonymousAccess bool, ctx *gin.Context, f models.QueryParam) (LogResp, error) {
+	var resp LogResp
+	var mu sync.Mutex
+	var wg sync.WaitGroup
+	var errs []error
+
+	for _, q := range f.Querys {
+		if !anonymousAccess && !CheckDsPerm(ctx, f.DatasourceId, f.Cate, q) {
+			return LogResp{}, fmt.Errorf("forbidden")
+		}
+
+		plug, exists := dscache.DsCache.Get(f.Cate, f.DatasourceId)
+		if !exists {
+			logger.Warningf("cluster:%d not exists query:%+v", f.DatasourceId, f)
+			return LogResp{}, fmt.Errorf("cluster not exists")
+		}
+
+		wg.Add(1)
+		go func(query interface{}) {
+			defer wg.Done()
+
+			data, total, err := plug.QueryLog(ctx.Request.Context(), query)
+			logger.Debugf("query log: req:%+v resp:%+v", query, data)
+			if err != nil {
+				errMsg := fmt.Sprintf("query data error: %v query:%v\n ", err, query)
+				logger.Warningf(errMsg)
+				mu.Lock()
+				errs = append(errs, err)
+				mu.Unlock()
+				return
+			}
+
+			mu.Lock()
+			resp.List = append(resp.List, data...)
+			resp.Total += total
+			mu.Unlock()
+		}(q)
+	}
+
+	wg.Wait()
+
+	if len(errs) > 0 {
+		return LogResp{}, errs[0]
+	}
+
+	if len(resp.List) == 0 {
+		return LogResp{}, fmt.Errorf("no data")
+	}
+
+	return resp, nil
 }
 
 func (rt *Router) QueryLogV2(c *gin.Context) {
 	var f models.QueryParam
 	ginx.BindJSON(c, &f)
 
-	var resp LogResp
-	var errMsg string
-	for _, q := range f.Querys {
-		if !rt.Center.AnonymousAccess.PromQuerier && !CheckDsPerm(c, f.DatasourceId, f.Cate, q) {
-			ginx.Bomb(200, "no permission")
-		}
-
-		plug, exists := dscache.DsCache.Get(f.Cate, f.DatasourceId)
-		if !exists {
-			logger.Warningf("cluster:%d not exists query:%+v", f.DatasourceId, f)
-			ginx.Bomb(200, "cluster not exists")
-		}
-
-		data, total, err := plug.QueryLog(c.Request.Context(), q)
-		if err != nil {
-			errMsg += fmt.Sprintf("query data error: %v query:%v\n ", err, q)
-			logger.Warningf("query data error: %v query:%v", err, q)
-			continue
-		}
-		resp.List = append(resp.List, data...)
-		resp.Total += total
-	}
-
-	if errMsg != "" || len(resp.List) == 0 {
-		ginx.Bomb(200, errMsg)
-	}
-
-	ginx.NewRender(c).Data(resp, nil)
+	resp, err := QueryLogConcurrently(rt.Center.AnonymousAccess.PromQuerier, c, f)
+	ginx.NewRender(c).Data(resp, err)
 }
 
 func (rt *Router) QueryLog(c *gin.Context) {
@@ -159,7 +244,7 @@ func (rt *Router) QueryLog(c *gin.Context) {
 	var resp []interface{}
 	for _, q := range f.Querys {
 		if !rt.Center.AnonymousAccess.PromQuerier && !CheckDsPerm(c, f.DatasourceId, f.Cate, q) {
-			ginx.Bomb(200, "no permission")
+			ginx.Bomb(200, "forbidden")
 		}
 
 		plug, exists := dscache.DsCache.Get("elasticsearch", f.DatasourceId)

center/router/router_recording_rule.go

@@ -6,10 +6,10 @@ import (
 	"time"
 
 	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 
 	"github.com/gin-gonic/gin"
 	"github.com/toolkits/pkg/ginx"
-	"github.com/toolkits/pkg/str"
 )
 
 func (rt *Router) recordingRuleGets(c *gin.Context) {
@@ -19,7 +19,7 @@ func (rt *Router) recordingRuleGets(c *gin.Context) {
 }
 
 func (rt *Router) recordingRuleGetsByGids(c *gin.Context) {
-	gids := str.IdsInt64(ginx.QueryStr(c, "gids", ""), ",")
+	gids := strx.IdsInt64ForAPI(ginx.QueryStr(c, "gids", ""), ",")
 	if len(gids) > 0 {
 		for _, gid := range gids {
 			rt.bgroCheck(c, gid)

center/router/router_source_token.go

@@ -0,0 +1,36 @@
+package router
+
+import (
+	"net/http"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/google/uuid"
+
+	"github.com/gin-gonic/gin"
+	"github.com/toolkits/pkg/ginx"
+)
+
+// sourceTokenAdd 生成新的源令牌
+func (rt *Router) sourceTokenAdd(c *gin.Context) {
+	var f models.SourceToken
+	ginx.BindJSON(c, &f)
+
+	if f.ExpireAt > 0 && f.ExpireAt <= time.Now().Unix() {
+		ginx.Bomb(http.StatusBadRequest, "expire time must be in the future")
+	}
+
+	token := uuid.New().String()
+
+	username := c.MustGet("username").(string)
+
+	f.Token = token
+	f.CreateBy = username
+	f.CreateAt = time.Now().Unix()
+
+	err := f.Add(rt.Ctx)
+	ginx.Dangerous(err)
+
+	go models.CleanupExpiredTokens(rt.Ctx)
+	ginx.NewRender(c).Data(token, nil)
+}

center/router/router_target.go

@@ -10,13 +10,13 @@ import (
 
 	"github.com/ccfos/nightingale/v6/models"
 	"github.com/ccfos/nightingale/v6/pkg/ctx"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 	"github.com/ccfos/nightingale/v6/storage"
 
 	"github.com/gin-gonic/gin"
 	"github.com/prometheus/common/model"
 	"github.com/toolkits/pkg/ginx"
 	"github.com/toolkits/pkg/logger"
-	"github.com/toolkits/pkg/str"
 )
 
 type TargetQuery struct {
@@ -44,7 +44,7 @@ func (rt *Router) targetGetsByHostFilter(c *gin.Context) {
 }
 
 func (rt *Router) targetGets(c *gin.Context) {
-	bgids := str.IdsInt64(ginx.QueryStr(c, "gids", ""), ",")
+	bgids := strx.IdsInt64ForAPI(ginx.QueryStr(c, "gids", ""), ",")
 	query := ginx.QueryStr(c, "query", "")
 	limit := ginx.QueryInt(c, "limit", 30)
 	downtime := ginx.QueryInt64(c, "downtime", 0)
@@ -56,7 +56,14 @@ func (rt *Router) targetGets(c *gin.Context) {
 	hosts := queryStrListField(c, "hosts", ",", " ", "\n")
 
 	var err error
-	if len(bgids) == 0 {
+	if len(bgids) > 0 {
+		// 如果用户当前查看的是未归组机器，会传入 bgids = [0]，此时是不需要校验的，故而排除这种情况
+		if !(len(bgids) == 1 && bgids[0] == 0) {
+			for _, gid := range bgids {
+				rt.bgroCheck(c, gid)
+			}
+		}
+	} else {
 		user := c.MustGet("user").(*models.User)
 		if !user.IsAdmin() {
 			// 如果是非 admin 用户，全部对象的情况，找到用户有权限的业务组
@@ -454,7 +461,7 @@ func (rt *Router) targetBindBgids(c *gin.Context) {
 			ginx.Dangerous(err)
 
 			if !can {
-				ginx.Bomb(http.StatusForbidden, "No permission. You are not admin of BG(%s)", bg.Name)
+				ginx.Bomb(http.StatusForbidden, "forbidden")
 			}
 		}
 		isNeverGrouped, checkErr := haveNeverGroupedIdent(rt.Ctx, f.Idents)
@@ -464,7 +471,7 @@ func (rt *Router) targetBindBgids(c *gin.Context) {
 			can, err := user.CheckPerm(rt.Ctx, "/targets/bind")
 			ginx.Dangerous(err)
 			if !can {
-				ginx.Bomb(http.StatusForbidden, "No permission. Only admin can assign BG")
+				ginx.Bomb(http.StatusForbidden, "forbidden")
 			}
 		}
 	}
@@ -549,7 +556,7 @@ func (rt *Router) checkTargetPerm(c *gin.Context, idents []string) {
 	ginx.Dangerous(err)
 
 	if len(nopri) > 0 {
-		ginx.Bomb(http.StatusForbidden, "No permission to operate the targets: %s", strings.Join(nopri, ", "))
+		ginx.Bomb(http.StatusForbidden, "forbidden")
 	}
 }
 
@@ -570,3 +577,27 @@ func (rt *Router) targetsOfAlertRule(c *gin.Context) {
 
 	ginx.NewRender(c).Data(ret, err)
 }
+
+func (rt *Router) checkTargetsExistByIndent(idents []string) {
+	notExists, err := models.TargetNoExistIdents(rt.Ctx, idents)
+	ginx.Dangerous(err)
+
+	if len(notExists) > 0 {
+		ginx.Bomb(http.StatusBadRequest, "targets not exist: %s", strings.Join(notExists, ", "))
+	}
+}
+
+func (rt *Router) targetsOfHostQuery(c *gin.Context) {
+	var queries []models.HostQuery
+	ginx.BindJSON(c, &queries)
+
+	hostsQuery := models.GetHostsQuery(queries)
+	session := models.TargetFilterQueryBuild(rt.Ctx, hostsQuery, 0, 0)
+	var lst []*models.Target
+	err := session.Find(&lst).Error
+	if err != nil {
+		ginx.Bomb(http.StatusInternalServerError, err.Error())
+	}
+
+	ginx.NewRender(c).Data(lst, nil)
+}

center/router/router_task.go

@@ -1,15 +1,16 @@
 package router
 
 import (
+	"strings"
 	"time"
 
 	"github.com/ccfos/nightingale/v6/alert/sender"
 	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 
 	"github.com/gin-gonic/gin"
 	"github.com/toolkits/pkg/ginx"
 	"github.com/toolkits/pkg/i18n"
-	"github.com/toolkits/pkg/str"
 )
 
 func (rt *Router) taskGets(c *gin.Context) {
@@ -40,7 +41,7 @@ func (rt *Router) taskGets(c *gin.Context) {
 }
 
 func (rt *Router) taskGetsByGids(c *gin.Context) {
-	gids := str.IdsInt64(ginx.QueryStr(c, "gids", ""), ",")
+	gids := strx.IdsInt64ForAPI(ginx.QueryStr(c, "gids", ""), ",")
 	if len(gids) > 0 {
 		for _, gid := range gids {
 			rt.bgroCheck(c, gid)
@@ -84,20 +85,6 @@ func (rt *Router) taskGetsByGids(c *gin.Context) {
 	}, nil)
 }
 
-type taskForm struct {
-	Title     string   `json:"title" binding:"required"`
-	Account   string   `json:"account" binding:"required"`
-	Batch     int      `json:"batch"`
-	Tolerance int      `json:"tolerance"`
-	Timeout   int      `json:"timeout"`
-	Pause     string   `json:"pause"`
-	Script    string   `json:"script" binding:"required"`
-	Args      string   `json:"args"`
-	Action    string   `json:"action" binding:"required"`
-	Creator   string   `json:"creator"`
-	Hosts     []string `json:"hosts" binding:"required"`
-}
-
 func (rt *Router) taskRecordAdd(c *gin.Context) {
 	var f *models.TaskRecord
 	ginx.BindJSON(c, &f)
@@ -112,11 +99,21 @@ func (rt *Router) taskAdd(c *gin.Context) {
 
 	var f models.TaskForm
 	ginx.BindJSON(c, &f)
+	// 把 f.Hosts 中的空字符串过滤掉
+	hosts := make([]string, 0, len(f.Hosts))
+	for i := range f.Hosts {
+		if strings.TrimSpace(f.Hosts[i]) != "" {
+			hosts = append(hosts, strings.TrimSpace(f.Hosts[i]))
+		}
+	}
+	f.Hosts = hosts
 
 	bgid := ginx.UrlParamInt64(c, "id")
 	user := c.MustGet("user").(*models.User)
 	f.Creator = user.Username
 
+	rt.checkTargetsExistByIndent(f.Hosts)
+
 	err := f.Verify()
 	ginx.Dangerous(err)
 

center/router/router_task_tpl.go

@@ -7,6 +7,7 @@ import (
 	"time"
 
 	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 
 	"github.com/gin-gonic/gin"
 	"github.com/toolkits/pkg/ginx"
@@ -35,7 +36,7 @@ func (rt *Router) taskTplGetsByGids(c *gin.Context) {
 	query := ginx.QueryStr(c, "query", "")
 	limit := ginx.QueryInt(c, "limit", 20)
 
-	gids := str.IdsInt64(ginx.QueryStr(c, "gids", ""), ",")
+	gids := strx.IdsInt64ForAPI(ginx.QueryStr(c, "gids", ""), ",")
 	if len(gids) > 0 {
 		for _, gid := range gids {
 			rt.bgroCheck(c, gid)
@@ -118,6 +119,18 @@ type taskTplForm struct {
 	Hosts     []string `json:"hosts"`
 }
 
+func (f *taskTplForm) Verify() {
+	// 传入的 f.Hosts 可能是 []string{"", "a", "b"}，需要过滤掉空字符串
+	args := make([]string, 0, len(f.Hosts))
+	for _, ident := range f.Hosts {
+		if strings.TrimSpace(ident) != "" {
+			args = append(args, strings.TrimSpace(ident))
+		}
+	}
+
+	f.Hosts = args
+}
+
 func (rt *Router) taskTplAdd(c *gin.Context) {
 	if !rt.Ibex.Enable {
 		ginx.Bomb(400, i18n.Sprintf(c.GetHeader("X-Language"), "This functionality has not been enabled. Please contact the system administrator to activate it."))
@@ -126,10 +139,13 @@ func (rt *Router) taskTplAdd(c *gin.Context) {
 
 	var f taskTplForm
 	ginx.BindJSON(c, &f)
+	f.Verify()
 
 	user := c.MustGet("user").(*models.User)
 	now := time.Now().Unix()
 
+	rt.checkTargetsExistByIndent(f.Hosts)
+
 	sort.Strings(f.Tags)
 
 	tpl := &models.TaskTpl{
@@ -167,6 +183,9 @@ func (rt *Router) taskTplPut(c *gin.Context) {
 
 	var f taskTplForm
 	ginx.BindJSON(c, &f)
+	f.Verify()
+
+	rt.checkTargetsExistByIndent(f.Hosts)
 
 	sort.Strings(f.Tags)
 

center/router/router_user.go

@@ -47,12 +47,27 @@ func (rt *Router) userGets(c *gin.Context) {
 	query := ginx.QueryStr(c, "query", "")
 	order := ginx.QueryStr(c, "order", "username")
 	desc := ginx.QueryBool(c, "desc", false)
+	usernames := strings.Split(ginx.QueryStr(c, "usernames", ""), ",")
+	phones := strings.Split(ginx.QueryStr(c, "phones", ""), ",")
+	emails := strings.Split(ginx.QueryStr(c, "emails", ""), ",")
+
+	if len(usernames) == 1 && usernames[0] == "" {
+		usernames = []string{}
+	}
+
+	if len(phones) == 1 && phones[0] == "" {
+		phones = []string{}
+	}
+
+	if len(emails) == 1 && emails[0] == "" {
+		emails = []string{}
+	}
 
 	go rt.UserCache.UpdateUsersLastActiveTime()
 	total, err := models.UserTotal(rt.Ctx, query, stime, etime)
 	ginx.Dangerous(err)
 
-	list, err := models.UserGets(rt.Ctx, query, limit, ginx.Offset(c, limit), stime, etime, order, desc)
+	list, err := models.UserGets(rt.Ctx, query, limit, ginx.Offset(c, limit), stime, etime, order, desc, usernames, phones, emails)
 	ginx.Dangerous(err)
 
 	user := c.MustGet("user").(*models.User)

center/router/router_user_group.go

@@ -6,11 +6,11 @@ import (
 
 	"github.com/ccfos/nightingale/v6/models"
 	"github.com/ccfos/nightingale/v6/pkg/flashduty"
+	"github.com/ccfos/nightingale/v6/pkg/strx"
 
 	"github.com/gin-gonic/gin"
 	"github.com/toolkits/pkg/ginx"
 	"github.com/toolkits/pkg/logger"
-	"github.com/toolkits/pkg/str"
 )
 
 func (rt *Router) checkBusiGroupPerm(c *gin.Context) {
@@ -32,7 +32,7 @@ func (rt *Router) userGroupGets(c *gin.Context) {
 }
 
 func (rt *Router) userGroupGetsByService(c *gin.Context) {
-	ids := str.IdsInt64(ginx.QueryStr(c, "ids", ""))
+	ids := strx.IdsInt64ForAPI(ginx.QueryStr(c, "ids", ""))
 
 	if len(ids) == 0 {
 		lst, err := models.UserGroupGetAll(rt.Ctx)
@@ -111,7 +111,6 @@ func (rt *Router) userGroupPut(c *gin.Context) {
 
 	me := c.MustGet("user").(*models.User)
 	ug := c.MustGet("user_group").(*models.UserGroup)
-	oldUGName := ug.Name
 
 	if ug.Name != f.Name {
 		// name changed, check duplication
@@ -130,7 +129,7 @@ func (rt *Router) userGroupPut(c *gin.Context) {
 	if f.IsSyncToFlashDuty || flashduty.NeedSyncTeam(rt.Ctx) {
 		ugs, err := flashduty.NewUserGroupSyncer(rt.Ctx, ug)
 		ginx.Dangerous(err)
-		err = ugs.SyncUGPut(oldUGName)
+		err = ugs.SyncUGPut()
 		ginx.Dangerous(err)
 	}
 	ginx.NewRender(c).Message(ug.Update(rt.Ctx, "Name", "Note", "UpdateAt", "UpdateBy"))
@@ -159,8 +158,11 @@ func (rt *Router) userGroupDel(c *gin.Context) {
 	if isSyncToFlashDuty || flashduty.NeedSyncTeam(rt.Ctx) {
 		ugs, err := flashduty.NewUserGroupSyncer(rt.Ctx, ug)
 		ginx.Dangerous(err)
-		err = ugs.SyncUGDel(ug.Name)
-		ginx.Dangerous(err)
+		err = ugs.SyncUGDel()
+		// 如果team 在 duty 被引用或者已经删除，会报错，可以忽略报错
+		if err != nil {
+			logger.Warningf("failed to sync user group %s to flashduty's team: %v", ug.Name, err)
+		}
 	}
 	ginx.NewRender(c).Message(ug.Del(rt.Ctx))
 

center/router/router_user_variable_config.go

@@ -40,7 +40,7 @@ func (rt *Router) userVariableConfigPut(context *gin.Context) {
 	user := context.MustGet("user").(*models.User)
 	if !user.IsAdmin() && f.CreateBy != user.Username {
 		// only admin or creator can update
-		ginx.Bomb(403, "no permission")
+		ginx.Bomb(403, "forbidden")
 	}
 
 	ginx.NewRender(context).Message(models.ConfigsUserVariableUpdate(rt.Ctx, f))
@@ -54,7 +54,7 @@ func (rt *Router) userVariableConfigDel(context *gin.Context) {
 	user := context.MustGet("user").(*models.User)
 	if !user.IsAdmin() && configs.CreateBy != user.Username {
 		// only admin or creator can delete
-		ginx.Bomb(403, "no permission")
+		ginx.Bomb(403, "forbidden")
 	}
 
 	if configs != nil && configs.External == models.ConfigExternal {

cmd/edge/edge.go

@@ -54,7 +54,7 @@ func Initialize(configDir string, cryptoKey string) (func(), error) {
 	targetCache := memsto.NewTargetCache(ctx, syncStats, redis)
 	busiGroupCache := memsto.NewBusiGroupCache(ctx, syncStats)
 	configCvalCache := memsto.NewCvalCache(ctx, syncStats)
-	idents := idents.New(ctx, redis)
+	idents := idents.New(ctx, redis, config.Pushgw)
 	metas := metas.New(redis)
 	writers := writer.NewWriters(config.Pushgw)
 	pushgwRouter := pushgwrt.New(config.HTTP, config.Pushgw, config.Alert, targetCache, busiGroupCache, idents, metas, writers, ctx)
@@ -74,6 +74,9 @@ func Initialize(configDir string, cryptoKey string) (func(), error) {
 		userCache := memsto.NewUserCache(ctx, syncStats)
 		userGroupCache := memsto.NewUserGroupCache(ctx, syncStats)
 		taskTplsCache := memsto.NewTaskTplCache(ctx)
+		notifyRuleCache := memsto.NewNotifyRuleCache(ctx, syncStats)
+		notifyChannelCache := memsto.NewNotifyChannelCache(ctx, syncStats)
+		messageTemplateCache := memsto.NewMessageTemplateCache(ctx, syncStats)
 
 		promClients := prom.NewPromClient(ctx)
 
@@ -82,7 +85,7 @@ func Initialize(configDir string, cryptoKey string) (func(), error) {
 		externalProcessors := process.NewExternalProcessors()
 
 		alert.Start(config.Alert, config.Pushgw, syncStats, alertStats, externalProcessors, targetCache, busiGroupCache, alertMuteCache,
-			alertRuleCache, notifyConfigCache, taskTplsCache, dsCache, ctx, promClients, userCache, userGroupCache)
+			alertRuleCache, notifyConfigCache, taskTplsCache, dsCache, ctx, promClients, userCache, userGroupCache, notifyRuleCache, notifyChannelCache, messageTemplateCache)
 
 		alertrtRouter := alertrt.New(config.HTTP, config.Alert, alertMuteCache, targetCache, busiGroupCache, alertStats, ctx, externalProcessors)
 

datasource/commons/eslike/eslike.go

@@ -19,20 +19,21 @@ import (
 )
 
 type Query struct {
-	Ref          string     `json:"ref" mapstructure:"ref"`
-	IndexType    string     `json:"index_type" mapstructure:"index_type"` // 普通索引:index 索引模式:index_pattern
-	Index        string     `json:"index" mapstructure:"index"`
-	IndexPattern string     `json:"index_pattern" mapstructure:"index_pattern"`
-	Filter       string     `json:"filter" mapstructure:"filter"`
-	MetricAggr   MetricAggr `json:"value" mapstructure:"value"`
-	GroupBy      []GroupBy  `json:"group_by" mapstructure:"group_by"`
-	DateField    string     `json:"date_field" mapstructure:"date_field"`
-	Interval     int64      `json:"interval" mapstructure:"interval"`
-	Start        int64      `json:"start" mapstructure:"start"`
-	End          int64      `json:"end" mapstructure:"end"`
-	P            int        `json:"page" mapstructure:"page"`           // 页码
-	Limit        int        `json:"limit" mapstructure:"limit"`         // 每页个数
-	Ascending    bool       `json:"ascending" mapstructure:"ascending"` // 按照DataField排序
+	Ref            string     `json:"ref" mapstructure:"ref"`
+	IndexType      string     `json:"index_type" mapstructure:"index_type"` // 普通索引:index 索引模式:index_pattern
+	Index          string     `json:"index" mapstructure:"index"`
+	IndexPatternId int64      `json:"index_pattern" mapstructure:"index_pattern"`
+	Filter         string     `json:"filter" mapstructure:"filter"`
+	Offset         int64      `json:"offset" mapstructure:"offset"`
+	MetricAggr     MetricAggr `json:"value" mapstructure:"value"`
+	GroupBy        []GroupBy  `json:"group_by" mapstructure:"group_by"`
+	DateField      string     `json:"date_field" mapstructure:"date_field"`
+	Interval       int64      `json:"interval" mapstructure:"interval"`
+	Start          int64      `json:"start" mapstructure:"start"`
+	End            int64      `json:"end" mapstructure:"end"`
+	P              int        `json:"page" mapstructure:"page"`           // 页码
+	Limit          int        `json:"limit" mapstructure:"limit"`         // 每页个数
+	Ascending      bool       `json:"ascending" mapstructure:"ascending"` // 按照DataField排序
 
 	Timeout  int `json:"timeout" mapstructure:"timeout"`
 	MaxShard int `json:"max_shard" mapstructure:"max_shard"`
@@ -344,13 +345,17 @@ func QueryData(ctx context.Context, queryParam interface{}, cliTimeout int64, ve
 
 	var indexArr []string
 	if param.IndexType == "index_pattern" {
-		indexArr = []string{param.IndexPattern}
-		if ip, ok := GetEsIndexPatternCacheType().Get(param.IndexPattern); ok {
+		if ip, ok := GetEsIndexPatternCacheType().Get(param.IndexPatternId); ok {
 			param.DateField = ip.TimeField
+			indexArr = []string{ip.Name}
+			param.Index = ip.Name
+		} else {
+			return nil, fmt.Errorf("index pattern:%d not found", param.IndexPatternId)
 		}
 	} else {
 		indexArr = strings.Split(param.Index, ",")
 	}
+
 	q := elastic.NewRangeQuery(param.DateField)
 	now := time.Now().Unix()
 	var start, end int64
@@ -368,6 +373,11 @@ func QueryData(ctx context.Context, queryParam interface{}, cliTimeout int64, ve
 		start = start - delay
 	}
 
+	if param.Offset > 0 {
+		end = end - param.Offset
+		start = start - param.Offset
+	}
+
 	q.Gte(time.Unix(start, 0).UnixMilli())
 	q.Lte(time.Unix(end, 0).UnixMilli())
 	q.Format("epoch_millis")
@@ -479,7 +489,7 @@ func QueryData(ctx context.Context, queryParam interface{}, cliTimeout int64, ve
 
 	source, _ := queryString.Source()
 	b, _ := json.Marshal(source)
-	logger.Debugf("query_data q:%+v tsAggr:%+v query_string:%s", param, tsAggr, string(b))
+	logger.Debugf("query_data q:%+v indexArr:%+v tsAggr:%+v query_string:%s", param, indexArr, tsAggr, string(b))
 
 	searchSource := elastic.NewSearchSource().
 		Query(queryString).
@@ -526,7 +536,16 @@ func QueryData(ctx context.Context, queryParam interface{}, cliTimeout int64, ve
 
 	GetBuckts("", keys, bucketsData, metrics, "", 0, param.MetricAggr.Func)
 
-	return TransferData(fmt.Sprintf("%s_%s", field, param.MetricAggr.Func), param.Ref, metrics.Data), nil
+	items, err := TransferData(fmt.Sprintf("%s_%s", field, param.MetricAggr.Func), param.Ref, metrics.Data), nil
+
+	var m map[string]interface{}
+	bs, _ := json.Marshal(queryParam)
+	json.Unmarshal(bs, &m)
+	m["index"] = param.Index
+	for i := range items {
+		items[i].Query = fmt.Sprintf("%+v", m)
+	}
+	return items, nil
 }
 
 func HitFilter(typ string) bool {
@@ -548,7 +567,17 @@ func QueryLog(ctx context.Context, queryParam interface{}, timeout int64, versio
 		param.Timeout = int(timeout)
 	}
 
-	indexArr := strings.Split(param.Index, ",")
+	var indexArr []string
+	if param.IndexType == "index_pattern" {
+		if ip, ok := GetEsIndexPatternCacheType().Get(param.IndexPatternId); ok {
+			param.DateField = ip.TimeField
+			indexArr = []string{ip.Name}
+		} else {
+			return nil, 0, fmt.Errorf("index pattern:%d not found", param.IndexPatternId)
+		}
+	} else {
+		indexArr = strings.Split(param.Index, ",")
+	}
 
 	now := time.Now().Unix()
 	var start, end int64

datasource/datasource.go

@@ -53,6 +53,20 @@ func init() {
 		PluginType:     "ck",
 		PluginTypeName: "ClickHouse",
 	}
+
+	DatasourceTypes[5] = DatasourceType{
+		Id:             5,
+		Category:       "timeseries",
+		PluginType:     "mysql",
+		PluginTypeName: "MySQL",
+	}
+
+	DatasourceTypes[6] = DatasourceType{
+		Id:             6,
+		Category:       "timeseries",
+		PluginType:     "pgsql",
+		PluginTypeName: "PostgreSQL",
+	}
 }
 
 type NewDatasrouceFn func(settings map[string]interface{}) (Datasource, error)
@@ -98,6 +112,7 @@ func GetDatasourceByType(typ string, settings map[string]interface{}) (Datasourc
 type DatasourceInfo struct {
 	Id             int64                  `json:"id"`
 	Name           string                 `json:"name"`
+	Identifier     string                 `json:"identifier"`
 	Description    string                 `json:"description"`
 	ClusterName    string                 `json:"cluster_name"`
 	Category       string                 `json:"category"`

datasource/es/es.go

@@ -187,6 +187,7 @@ func (e *Elasticsearch) QueryData(ctx context.Context, queryParam interface{}) (
 	search := func(ctx context.Context, indices []string, source interface{}, timeout int, maxShard int) (*elastic.SearchResult, error) {
 		return e.Client.Search().
 			Index(indices...).
+			IgnoreUnavailable(true).
 			Source(source).
 			Timeout(fmt.Sprintf("%ds", timeout)).
 			MaxConcurrentShardRequests(maxShard).
@@ -204,7 +205,7 @@ func (e *Elasticsearch) QueryIndices() ([]string, error) {
 
 func (e *Elasticsearch) QueryFields(indexs []string) ([]string, error) {
 	var fields []string
-	result, err := elastic.NewGetFieldMappingService(e.Client).Index(indexs...).Do(context.Background())
+	result, err := elastic.NewGetFieldMappingService(e.Client).Index(indexs...).IgnoreUnavailable(true).Do(context.Background())
 	if err != nil {
 		return fields, err
 	}
@@ -264,6 +265,7 @@ func (e *Elasticsearch) QueryLog(ctx context.Context, queryParam interface{}) ([
 
 		return e.Client.Search().
 			Index(indices...).
+			IgnoreUnavailable(true).
 			MaxConcurrentShardRequests(maxShard).
 			Source(source).
 			Timeout(fmt.Sprintf("%ds", timeout)).
@@ -276,6 +278,7 @@ func (e *Elasticsearch) QueryLog(ctx context.Context, queryParam interface{}) ([
 func (e *Elasticsearch) QueryFieldValue(indexs []string, field string, query string) ([]string, error) {
 	var values []string
 	search := e.Client.Search().
+		IgnoreUnavailable(true).
 		Index(indexs...).
 		Size(0)
 
@@ -359,6 +362,7 @@ func (e *Elasticsearch) QueryMapData(ctx context.Context, query interface{}) ([]
 
 		return e.Client.Search().
 			Index(indices...).
+			IgnoreUnavailable(true).
 			Source(source).
 			Timeout(fmt.Sprintf("%ds", timeout)).
 			Do(ctx)

datasource/mysql/mysql.go

@@ -0,0 +1,227 @@
+package mysql
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/datasource"
+	"github.com/ccfos/nightingale/v6/dskit/mysql"
+	"github.com/ccfos/nightingale/v6/dskit/sqlbase"
+	"github.com/ccfos/nightingale/v6/dskit/types"
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/ccfos/nightingale/v6/pkg/macros"
+
+	"github.com/mitchellh/mapstructure"
+	"github.com/toolkits/pkg/logger"
+)
+
+const (
+	MySQLType = "mysql"
+)
+
+func init() {
+	datasource.RegisterDatasource(MySQLType, new(MySQL))
+}
+
+type MySQL struct {
+	mysql.MySQL `json:",inline" mapstructure:",squash"`
+}
+
+type QueryParam struct {
+	Ref      string          `json:"ref" mapstructure:"ref"`
+	Database string          `json:"database" mapstructure:"database"`
+	Table    string          `json:"table" mapstructure:"table"`
+	SQL      string          `json:"sql" mapstructure:"sql"`
+	Keys     datasource.Keys `json:"keys" mapstructure:"keys"`
+	From     int64           `json:"from" mapstructure:"from"`
+	To       int64           `json:"to" mapstructure:"to"`
+}
+
+func (m *MySQL) InitClient() error {
+	if len(m.Shards) == 0 {
+		return fmt.Errorf("not found mysql addr, please check datasource config")
+	}
+	if _, err := m.NewConn(context.TODO(), ""); err != nil {
+		return err
+	}
+	return nil
+}
+
+func (m *MySQL) Init(settings map[string]interface{}) (datasource.Datasource, error) {
+	newest := new(MySQL)
+	err := mapstructure.Decode(settings, newest)
+	return newest, err
+}
+
+func (m *MySQL) Validate(ctx context.Context) error {
+	if len(m.Shards) == 0 || len(strings.TrimSpace(m.Shards[0].Addr)) == 0 {
+		return fmt.Errorf("mysql addr is invalid, please check datasource setting")
+	}
+
+	if len(strings.TrimSpace(m.Shards[0].User)) == 0 {
+		return fmt.Errorf("mysql user is invalid, please check datasource setting")
+	}
+
+	return nil
+}
+
+// Equal compares whether two objects are the same, used for caching
+func (m *MySQL) Equal(p datasource.Datasource) bool {
+	newest, ok := p.(*MySQL)
+	if !ok {
+		logger.Errorf("unexpected plugin type, expected is mysql")
+		return false
+	}
+
+	if len(m.Shards) == 0 || len(newest.Shards) == 0 {
+		return false
+	}
+
+	oldShard := m.Shards[0]
+	newShard := newest.Shards[0]
+
+	if oldShard.Addr != newShard.Addr {
+		return false
+	}
+
+	if oldShard.User != newShard.User {
+		return false
+	}
+
+	if oldShard.Password != newShard.Password {
+		return false
+	}
+
+	if oldShard.MaxQueryRows != newShard.MaxQueryRows {
+		return false
+	}
+
+	if oldShard.Timeout != newShard.Timeout {
+		return false
+	}
+
+	if oldShard.MaxIdleConns != newShard.MaxIdleConns {
+		return false
+	}
+
+	if oldShard.MaxOpenConns != newShard.MaxOpenConns {
+		return false
+	}
+
+	if oldShard.ConnMaxLifetime != newShard.ConnMaxLifetime {
+		return false
+	}
+
+	return true
+}
+
+func (m *MySQL) MakeLogQuery(ctx context.Context, query interface{}, eventTags []string, start, end int64) (interface{}, error) {
+	return nil, nil
+}
+
+func (m *MySQL) MakeTSQuery(ctx context.Context, query interface{}, eventTags []string, start, end int64) (interface{}, error) {
+	return nil, nil
+}
+
+func (m *MySQL) QueryMapData(ctx context.Context, query interface{}) ([]map[string]string, error) {
+	return nil, nil
+}
+
+func (m *MySQL) QueryData(ctx context.Context, query interface{}) ([]models.DataResp, error) {
+	mysqlQueryParam := new(QueryParam)
+	if err := mapstructure.Decode(query, mysqlQueryParam); err != nil {
+		return nil, err
+	}
+
+	if strings.Contains(mysqlQueryParam.SQL, "$__") {
+		var err error
+		mysqlQueryParam.SQL, err = macros.Macro(mysqlQueryParam.SQL, mysqlQueryParam.From, mysqlQueryParam.To)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	if mysqlQueryParam.Keys.ValueKey == "" {
+		return nil, fmt.Errorf("valueKey is required")
+	}
+
+	timeout := m.Shards[0].Timeout
+	if timeout == 0 {
+		timeout = 60
+	}
+
+	timeoutCtx, cancel := context.WithTimeout(ctx, time.Duration(timeout)*time.Second)
+	defer cancel()
+
+	items, err := m.QueryTimeseries(timeoutCtx, &sqlbase.QueryParam{
+		Sql: mysqlQueryParam.SQL,
+		Keys: types.Keys{
+			ValueKey: mysqlQueryParam.Keys.ValueKey,
+			LabelKey: mysqlQueryParam.Keys.LabelKey,
+			TimeKey:  mysqlQueryParam.Keys.TimeKey,
+		},
+	})
+
+	if err != nil {
+		logger.Warningf("query:%+v get data err:%v", mysqlQueryParam, err)
+		return []models.DataResp{}, err
+	}
+	data := make([]models.DataResp, 0)
+	for i := range items {
+		data = append(data, models.DataResp{
+			Ref:    mysqlQueryParam.Ref,
+			Metric: items[i].Metric,
+			Values: items[i].Values,
+		})
+	}
+
+	return data, nil
+}
+
+func (m *MySQL) QueryLog(ctx context.Context, query interface{}) ([]interface{}, int64, error) {
+	mysqlQueryParam := new(QueryParam)
+	if err := mapstructure.Decode(query, mysqlQueryParam); err != nil {
+		return nil, 0, err
+	}
+
+	if strings.Contains(mysqlQueryParam.SQL, "$__") {
+		var err error
+		mysqlQueryParam.SQL, err = macros.Macro(mysqlQueryParam.SQL, mysqlQueryParam.From, mysqlQueryParam.To)
+		if err != nil {
+			return nil, 0, err
+		}
+	}
+
+	timeout := m.Shards[0].Timeout
+	if timeout == 0 {
+		timeout = 60
+	}
+
+	timeoutCtx, cancel := context.WithTimeout(ctx, time.Duration(timeout)*time.Second)
+	defer cancel()
+
+	items, err := m.Query(timeoutCtx, &sqlbase.QueryParam{
+		Sql: mysqlQueryParam.SQL,
+	})
+
+	if err != nil {
+		logger.Warningf("query:%+v get data err:%v", mysqlQueryParam, err)
+		return []interface{}{}, 0, err
+	}
+	logs := make([]interface{}, 0)
+	for i := range items {
+		logs = append(logs, items[i])
+	}
+
+	return logs, 0, nil
+}
+
+func (m *MySQL) DescribeTable(ctx context.Context, query interface{}) ([]*types.ColumnProperty, error) {
+	mysqlQueryParam := new(QueryParam)
+	if err := mapstructure.Decode(query, mysqlQueryParam); err != nil {
+		return nil, err
+	}
+	return m.DescTable(ctx, mysqlQueryParam.Database, mysqlQueryParam.Table)
+}

datasource/postgresql/postgresql.go

@@ -0,0 +1,346 @@
+package postgresql
+
+import (
+	"context"
+	"fmt"
+	"regexp"
+	"strings"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/datasource"
+	"github.com/ccfos/nightingale/v6/pkg/macros"
+
+	"github.com/ccfos/nightingale/v6/dskit/postgres"
+	"github.com/ccfos/nightingale/v6/dskit/sqlbase"
+	"github.com/ccfos/nightingale/v6/dskit/types"
+	"github.com/ccfos/nightingale/v6/models"
+	"github.com/mitchellh/mapstructure"
+	"github.com/toolkits/pkg/logger"
+)
+
+const (
+	PostgreSQLType = "pgsql"
+)
+
+var (
+	regx = "(?i)from\\s+([a-zA-Z0-9_]+)\\.([a-zA-Z0-9_]+)\\.([a-zA-Z0-9_]+)"
+)
+
+func init() {
+	datasource.RegisterDatasource(PostgreSQLType, new(PostgreSQL))
+}
+
+type PostgreSQL struct {
+	Shards []*postgres.PostgreSQL `json:"pgsql.shards" mapstructure:"pgsql.shards"`
+}
+
+type QueryParam struct {
+	Ref      string          `json:"ref" mapstructure:"ref"`
+	Database string          `json:"database" mapstructure:"database"`
+	Table    string          `json:"table" mapstructure:"table"`
+	SQL      string          `json:"sql" mapstructure:"sql"`
+	Keys     datasource.Keys `json:"keys" mapstructure:"keys"`
+	From     int64           `json:"from" mapstructure:"from"`
+	To       int64           `json:"to" mapstructure:"to"`
+}
+
+func (p *PostgreSQL) InitClient() error {
+	if len(p.Shards) == 0 {
+		return fmt.Errorf("not found postgresql addr, please check datasource config")
+	}
+	for _, shard := range p.Shards {
+		if db, err := shard.NewConn(context.TODO(), "postgres"); err != nil {
+			defer sqlbase.CloseDB(db)
+			return err
+		}
+	}
+	return nil
+}
+
+func (p *PostgreSQL) Init(settings map[string]interface{}) (datasource.Datasource, error) {
+	newest := new(PostgreSQL)
+	err := mapstructure.Decode(settings, newest)
+	return newest, err
+}
+
+func (p *PostgreSQL) Validate(ctx context.Context) error {
+	if len(p.Shards) == 0 || len(strings.TrimSpace(p.Shards[0].Addr)) == 0 {
+		return fmt.Errorf("postgresql addr is invalid, please check datasource setting")
+	}
+
+	if len(strings.TrimSpace(p.Shards[0].User)) == 0 {
+		return fmt.Errorf("postgresql user is invalid, please check datasource setting")
+	}
+
+	return nil
+}
+
+// Equal compares whether two objects are the same, used for caching
+func (p *PostgreSQL) Equal(d datasource.Datasource) bool {
+	newest, ok := d.(*PostgreSQL)
+	if !ok {
+		logger.Errorf("unexpected plugin type, expected is postgresql")
+		return false
+	}
+
+	if len(p.Shards) == 0 || len(newest.Shards) == 0 {
+		return false
+	}
+
+	oldShard := p.Shards[0]
+	newShard := newest.Shards[0]
+
+	if oldShard.Addr != newShard.Addr {
+		return false
+	}
+
+	if oldShard.User != newShard.User {
+		return false
+	}
+
+	if oldShard.Password != newShard.Password {
+		return false
+	}
+
+	if oldShard.MaxQueryRows != newShard.MaxQueryRows {
+		return false
+	}
+
+	if oldShard.Timeout != newShard.Timeout {
+		return false
+	}
+
+	if oldShard.MaxIdleConns != newShard.MaxIdleConns {
+		return false
+	}
+
+	if oldShard.MaxOpenConns != newShard.MaxOpenConns {
+		return false
+	}
+
+	if oldShard.ConnMaxLifetime != newShard.ConnMaxLifetime {
+		return false
+	}
+
+	return true
+}
+
+func (p *PostgreSQL) ShowDatabases(ctx context.Context) ([]string, error) {
+	return p.Shards[0].ShowDatabases(ctx, "")
+}
+
+func (p *PostgreSQL) ShowTables(ctx context.Context, database string) ([]string, error) {
+	p.Shards[0].DB = database
+	rets, err := p.Shards[0].ShowTables(ctx, "")
+	if err != nil {
+		return nil, err
+	}
+	tables := make([]string, 0, len(rets))
+	for scheme, tabs := range rets {
+		for _, tab := range tabs {
+			tables = append(tables, scheme+"."+tab)
+		}
+	}
+	return tables, nil
+}
+
+func (p *PostgreSQL) MakeLogQuery(ctx context.Context, query interface{}, eventTags []string, start, end int64) (interface{}, error) {
+	return nil, nil
+}
+
+func (p *PostgreSQL) MakeTSQuery(ctx context.Context, query interface{}, eventTags []string, start, end int64) (interface{}, error) {
+	return nil, nil
+}
+
+func (p *PostgreSQL) QueryMapData(ctx context.Context, query interface{}) ([]map[string]string, error) {
+	return nil, nil
+}
+
+func (p *PostgreSQL) QueryData(ctx context.Context, query interface{}) ([]models.DataResp, error) {
+	postgresqlQueryParam := new(QueryParam)
+	if err := mapstructure.Decode(query, postgresqlQueryParam); err != nil {
+		return nil, err
+	}
+
+	if strings.Contains(postgresqlQueryParam.SQL, "$__") {
+		var err error
+		postgresqlQueryParam.SQL, err = macros.Macro(postgresqlQueryParam.SQL, postgresqlQueryParam.From, postgresqlQueryParam.To)
+		if err != nil {
+			return nil, err
+		}
+	}
+	if postgresqlQueryParam.Database != "" {
+		p.Shards[0].DB = postgresqlQueryParam.Database
+	} else {
+		db, err := parseDBName(postgresqlQueryParam.SQL)
+		if err != nil {
+			return nil, err
+		}
+		p.Shards[0].DB = db
+	}
+
+	timeout := p.Shards[0].Timeout
+	if timeout == 0 {
+		timeout = 60
+	}
+	timeoutCtx, cancel := context.WithTimeout(ctx, time.Duration(timeout)*time.Second)
+	defer cancel()
+
+	items, err := p.Shards[0].QueryTimeseries(timeoutCtx, &sqlbase.QueryParam{
+		Sql: postgresqlQueryParam.SQL,
+		Keys: types.Keys{
+			ValueKey: postgresqlQueryParam.Keys.ValueKey,
+			LabelKey: postgresqlQueryParam.Keys.LabelKey,
+			TimeKey:  postgresqlQueryParam.Keys.TimeKey,
+		},
+	})
+
+	if err != nil {
+		logger.Warningf("query:%+v get data err:%v", postgresqlQueryParam, err)
+		return []models.DataResp{}, err
+	}
+	data := make([]models.DataResp, 0)
+	for i := range items {
+		data = append(data, models.DataResp{
+			Ref:    postgresqlQueryParam.Ref,
+			Metric: items[i].Metric,
+			Values: items[i].Values,
+		})
+	}
+
+	// parse resp to time series data
+	logger.Infof("req:%+v keys:%+v \n data:%v", postgresqlQueryParam, postgresqlQueryParam.Keys, data)
+
+	return data, nil
+}
+
+func (p *PostgreSQL) QueryLog(ctx context.Context, query interface{}) ([]interface{}, int64, error) {
+	postgresqlQueryParam := new(QueryParam)
+	if err := mapstructure.Decode(query, postgresqlQueryParam); err != nil {
+		return nil, 0, err
+	}
+	if postgresqlQueryParam.Database != "" {
+		p.Shards[0].DB = postgresqlQueryParam.Database
+	} else {
+		db, err := parseDBName(postgresqlQueryParam.SQL)
+		if err != nil {
+			return nil, 0, err
+		}
+		p.Shards[0].DB = db
+	}
+
+	if strings.Contains(postgresqlQueryParam.SQL, "$__") {
+		var err error
+		postgresqlQueryParam.SQL, err = macros.Macro(postgresqlQueryParam.SQL, postgresqlQueryParam.From, postgresqlQueryParam.To)
+		if err != nil {
+			return nil, 0, err
+		}
+	}
+
+	timeout := p.Shards[0].Timeout
+	if timeout == 0 {
+		timeout = 60
+	}
+	timeoutCtx, cancel := context.WithTimeout(ctx, time.Duration(timeout)*time.Second)
+	defer cancel()
+	items, err := p.Shards[0].Query(timeoutCtx, &sqlbase.QueryParam{
+		Sql: postgresqlQueryParam.SQL,
+	})
+	if err != nil {
+		logger.Warningf("query:%+v get data err:%v", postgresqlQueryParam, err)
+		return []interface{}{}, 0, err
+	}
+	logs := make([]interface{}, 0)
+	for i := range items {
+		logs = append(logs, items[i])
+	}
+
+	return logs, 0, nil
+}
+
+func (p *PostgreSQL) DescribeTable(ctx context.Context, query interface{}) ([]*types.ColumnProperty, error) {
+	postgresqlQueryParam := new(QueryParam)
+	if err := mapstructure.Decode(query, postgresqlQueryParam); err != nil {
+		return nil, err
+	}
+	p.Shards[0].DB = postgresqlQueryParam.Database
+	pairs := strings.Split(postgresqlQueryParam.Table, ".") // format: scheme.table_name
+	scheme := ""
+	table := postgresqlQueryParam.Table
+	if len(pairs) == 2 {
+		scheme = pairs[0]
+		table = pairs[1]
+	}
+	return p.Shards[0].DescTable(ctx, scheme, table)
+}
+
+func parseDBName(sql string) (db string, err error) {
+	re := regexp.MustCompile(regx)
+	matches := re.FindStringSubmatch(sql)
+	if len(matches) != 4 {
+		return "", fmt.Errorf("no valid table name in format database.schema.table found")
+	}
+	return matches[1], nil
+}
+
+func extractColumns(sql string) ([]string, error) {
+	// 将 SQL 转换为小写以简化匹配
+	sql = strings.ToLower(sql)
+
+	// 匹配 SELECT 和 FROM 之间的内容
+	re := regexp.MustCompile(`select\s+(.*?)\s+from`)
+	matches := re.FindStringSubmatch(sql)
+
+	if len(matches) < 2 {
+		return nil, fmt.Errorf("no columns found or invalid SQL syntax")
+	}
+
+	// 提取列部分
+	columnsString := matches[1]
+
+	// 分割列
+	columns := splitColumns(columnsString)
+
+	// 清理每个列名
+	for i, col := range columns {
+		columns[i] = strings.TrimSpace(col)
+	}
+
+	return columns, nil
+}
+
+func splitColumns(columnsString string) []string {
+	var columns []string
+	var currentColumn strings.Builder
+	parenthesesCount := 0
+	inQuotes := false
+
+	for _, char := range columnsString {
+		switch char {
+		case '(':
+			parenthesesCount++
+			currentColumn.WriteRune(char)
+		case ')':
+			parenthesesCount--
+			currentColumn.WriteRune(char)
+		case '\'', '"':
+			inQuotes = !inQuotes
+			currentColumn.WriteRune(char)
+		case ',':
+			if parenthesesCount == 0 && !inQuotes {
+				columns = append(columns, currentColumn.String())
+				currentColumn.Reset()
+			} else {
+				currentColumn.WriteRune(char)
+			}
+		default:
+			currentColumn.WriteRune(char)
+		}
+	}
+
+	if currentColumn.Len() > 0 {
+		columns = append(columns, currentColumn.String())
+	}
+
+	return columns
+}

docker/compose-bridge/etc-nightingale/config.toml

@@ -5,7 +5,7 @@ RunMode = "release"
 # log write dir
 Dir = "logs"
 # log level: DEBUG INFO WARNING ERROR
-Level = "DEBUG"
+Level = "INFO"
 # stdout, stderr, file
 Output = "stdout"
 # # rotate by time
@@ -75,15 +75,6 @@ DefaultRoles = ["Standard"]
 [HTTP.RSA]
 # open RSA
 OpenRSA = false
-# Before replacing the key file, make sure that there are no encrypted variables in the database "configs".
-# It is recommended to decrypt and remove all encrypted values from the database before replacing the key file.
-# This will prevent any potential issues with accessing or decrypting the variables using the new key file.
-# RSA public key (auto carete)
-RSAPublicKeyPath = "etc/rsa/public.pem"
-# RSA private key (auto carete)
-RSAPrivateKeyPath = "etc/rsa/private.pem"
-# RSA private key password
-RSAPassWord = "n9e@n9e!"
 
 [DB]
 # postgres: DSN="host=127.0.0.1 port=5432 user=root dbname=n9e_v6 password=1234 sslmode=disable"
@@ -138,8 +129,6 @@ AlertDetail = false
 [Pushgw]
 # use target labels in database instead of in series
 LabelRewrite = true
-# # default busigroup key name
-# BusiGroupLabelKey = "busigroup"
 ForceUseServerTS = true
 
 # [Pushgw.DebugSample]

docker/compose-host-network-metric-log/etc-nightingale/config.toml

@@ -5,7 +5,7 @@ RunMode = "release"
 # log write dir
 Dir = "logs"
 # log level: DEBUG INFO WARNING ERROR
-Level = "DEBUG"
+Level = "INFO"
 # stdout, stderr, file
 Output = "file"
 # # rotate by time
@@ -71,15 +71,6 @@ DefaultRoles = ["Standard"]
 [HTTP.RSA]
 # open RSA
 OpenRSA = false
-# Before replacing the key file, make sure that there are no encrypted variables in the database "configs".
-# It is recommended to decrypt and remove all encrypted values from the database before replacing the key file.
-# This will prevent any potential issues with accessing or decrypting the variables using the new key file.
-# RSA public key (auto carete)
-RSAPublicKeyPath = "etc/rsa/public.pem"
-# RSA private key (auto carete)
-RSAPrivateKeyPath = "etc/rsa/private.pem"
-# RSA private key password
-RSAPassWord = "n9e@n9e!"
 
 [DB]
 # postgres: host=%s port=%s user=%s dbname=%s password=%s sslmode=%s
@@ -135,8 +126,6 @@ AlertDetail = true
 [Pushgw]
 # use target labels in database instead of in series
 LabelRewrite = true
-# # default busigroup key name
-# BusiGroupLabelKey = "busigroup"
 ForceUseServerTS = true
 
 # [Pushgw.DebugSample]

docker/compose-host-network/etc-nightingale/config.toml

@@ -5,7 +5,7 @@ RunMode = "release"
 # log write dir
 Dir = "logs"
 # log level: DEBUG INFO WARNING ERROR
-Level = "DEBUG"
+Level = "INFO"
 # stdout, stderr, file
 Output = "stdout"
 # # rotate by time
@@ -71,15 +71,6 @@ DefaultRoles = ["Standard"]
 [HTTP.RSA]
 # open RSA
 OpenRSA = false
-# Before replacing the key file, make sure that there are no encrypted variables in the database "configs".
-# It is recommended to decrypt and remove all encrypted values from the database before replacing the key file.
-# This will prevent any potential issues with accessing or decrypting the variables using the new key file.
-# RSA public key (auto carete)
-RSAPublicKeyPath = "etc/rsa/public.pem"
-# RSA private key (auto carete)
-RSAPrivateKeyPath = "etc/rsa/private.pem"
-# RSA private key password
-RSAPassWord = "n9e@n9e!"
 
 [DB]
 # postgres: host=%s port=%s user=%s dbname=%s password=%s sslmode=%s
@@ -135,8 +126,6 @@ AlertDetail = true
 [Pushgw]
 # use target labels in database instead of in series
 LabelRewrite = true
-# # default busigroup key name
-# BusiGroupLabelKey = "busigroup"
 ForceUseServerTS = true
 
 # [Pushgw.DebugSample]

docker/compose-postgres/initsql_for_postgres/a-n9e-for-Postgres.sql

@@ -204,6 +204,7 @@ CREATE TABLE board (
     public smallint not null default 0 ,
     built_in smallint not null default 0 ,
     hide smallint not null default 0 ,
+    public_cate bigint NOT NULL DEFAULT 0,
     create_at bigint not null default 0,
     create_by varchar(64) not null default '',
     update_at bigint not null default 0,
@@ -217,6 +218,7 @@ COMMENT ON COLUMN board.tags IS 'split by space';
 COMMENT ON COLUMN board.public IS '0:false 1:true';
 COMMENT ON COLUMN board.built_in IS '0:false 1:true';
 COMMENT ON COLUMN board.hide IS '0:false 1:true';
+COMMENT ON COLUMN board.public_cate IS '0 anonymous 1 login 2 busi';
 
 
 -- for dashboard new version
@@ -429,43 +431,31 @@ CREATE TABLE target (
     ident varchar(191) not null,
     note varchar(255) not null default '',
     tags varchar(512) not null default '',
+    host_tags text,
     host_ip varchar(15) default '',
     agent_version varchar(255) default '',
     engine_name varchar(255) default '',
+    os varchar(31) default '',
     update_at bigint not null default 0,
     PRIMARY KEY (id),
     UNIQUE (ident)
 );
 
 CREATE INDEX ON target (group_id);
+CREATE INDEX idx_host_ip ON target (host_ip);
+CREATE INDEX idx_agent_version ON target (agent_version);
+CREATE INDEX idx_engine_name ON target (engine_name);
+CREATE INDEX idx_os ON target (os);
 
 COMMENT ON COLUMN target.group_id IS 'busi group id';
 COMMENT ON COLUMN target.ident IS 'target id';
 COMMENT ON COLUMN target.note IS 'append to alert event as field';
 COMMENT ON COLUMN target.tags IS 'append to series data as tags, split by space, append external space at suffix';
+COMMENT ON COLUMN target.host_tags IS 'global labels set in conf file';
 COMMENT ON COLUMN target.host_ip IS 'IPv4 string';
 COMMENT ON COLUMN target.agent_version IS 'agent version';
 COMMENT ON COLUMN target.engine_name IS 'engine_name';
--- case1: target_idents; case2: target_tags
--- CREATE TABLE collect_rule (
---     id bigserial,
---     group_id bigint not null default 0 comment 'busi group id',
---     cluster varchar(128) not null,
---     target_idents varchar(512) not null default '' comment 'ident list, split by space',
---     target_tags varchar(512) not null default '' comment 'filter targets by tags, split by space',
---     name varchar(191) not null default '',
---     note varchar(255) not null default '',
---     step int not null,
---     type varchar(64) not null comment 'e.g. port proc log plugin',
---     data text not null,
---     append_tags varchar(255) not null default '' comment 'split by space: e.g. mod=n9e dept=cloud',
---     create_at bigint not null default 0,
---     create_by varchar(64) not null default '',
---     update_at bigint not null default 0,
---     update_by varchar(64) not null default '',
---     PRIMARY KEY (id),
---     KEY (group_id, type, name)
--- ) ;
+COMMENT ON COLUMN target.os IS 'os type';
 
 CREATE TABLE metric_view (
     id bigserial,
@@ -734,6 +724,7 @@ CREATE TABLE datasource
 (
     id serial,
     name varchar(191) not null default '',
+    identifier varchar(255) not null default '',
     description varchar(255) not null default '',
     category varchar(255) not null default '',
     plugin_id int  not null default 0,
@@ -751,8 +742,8 @@ CREATE TABLE datasource
     updated_by varchar(64) not null default '',
     UNIQUE (name),
     PRIMARY KEY (id)
-) ; 
-  
+) ;
+
 CREATE TABLE builtin_cate (
     id bigserial,
     name varchar(191) not null,
@@ -795,10 +786,12 @@ CREATE TABLE es_index_pattern (
     create_by varchar(64) default '',
     update_at bigint default '0',
     update_by varchar(64) default '',
+    note varchar(4096) not null default '',
     PRIMARY KEY (id),
     UNIQUE (datasource_id, name)
 ) ;
 COMMENT ON COLUMN es_index_pattern.datasource_id IS 'datasource id';
+COMMENT ON COLUMN es_index_pattern.note IS 'description of metric in Chinese';
 
 CREATE TABLE builtin_metrics (
   id bigserial,
@@ -813,6 +806,7 @@ CREATE TABLE builtin_metrics (
   created_by varchar(191) NOT NULL DEFAULT '',
   updated_at bigint NOT NULL DEFAULT 0,
   updated_by varchar(191) NOT NULL DEFAULT '',
+  uuid BIGINT NOT NULL DEFAULT 0,
   PRIMARY KEY (id),
   UNIQUE (lang, collector, typ, name)
 );
@@ -834,6 +828,7 @@ COMMENT ON COLUMN builtin_metrics.created_at IS 'create time';
 COMMENT ON COLUMN builtin_metrics.created_by IS 'creator';
 COMMENT ON COLUMN builtin_metrics.updated_at IS 'update time';
 COMMENT ON COLUMN builtin_metrics.updated_by IS 'updater';
+COMMENT ON COLUMN builtin_metrics.uuid IS 'unique identifier';
 
 CREATE TABLE metric_filter (
   id BIGSERIAL PRIMARY KEY,
@@ -903,4 +898,128 @@ CREATE TABLE dash_annotation (
     create_by varchar(64) not null default '',
     update_at bigint not null default 0,
     update_by varchar(64) not null default ''
+);
+
+CREATE TABLE source_token (
+    id bigserial PRIMARY KEY,
+    source_type varchar(64) NOT NULL DEFAULT '',
+    source_id varchar(255) NOT NULL DEFAULT '',
+    token varchar(255) NOT NULL DEFAULT '',
+    expire_at bigint NOT NULL DEFAULT 0,
+    create_at bigint NOT NULL DEFAULT 0,
+    create_by varchar(64) NOT NULL DEFAULT ''
+);
+
+CREATE INDEX idx_source_token_type_id_token ON source_token (source_type, source_id, token);
+
+CREATE TABLE notification_record (
+    id BIGSERIAL PRIMARY KEY,
+    notify_rule_id BIGINT NOT NULL DEFAULT 0,
+    event_id bigint NOT NULL,
+    sub_id bigint DEFAULT NULL,
+    channel varchar(255) NOT NULL,
+    status bigint DEFAULT NULL,
+    target varchar(1024) NOT NULL,
+    details varchar(2048) DEFAULT '',
+    created_at bigint NOT NULL
+);
+
+CREATE INDEX idx_evt ON notification_record (event_id);
+
+COMMENT ON COLUMN notification_record.event_id IS 'event history id';
+COMMENT ON COLUMN notification_record.sub_id IS 'subscribed rule id';
+COMMENT ON COLUMN notification_record.channel IS 'notification channel name';
+COMMENT ON COLUMN notification_record.status IS 'notification status';
+COMMENT ON COLUMN notification_record.target IS 'notification target';
+COMMENT ON COLUMN notification_record.details IS 'notification other info';
+COMMENT ON COLUMN notification_record.created_at IS 'create time';
+
+CREATE TABLE target_busi_group (
+    id BIGSERIAL PRIMARY KEY,
+    target_ident varchar(191) NOT NULL,
+    group_id bigint NOT NULL,
+    update_at bigint NOT NULL
+);
+
+CREATE UNIQUE INDEX idx_target_group ON target_busi_group (target_ident, group_id);
+
+CREATE TABLE user_token (
+    id BIGSERIAL PRIMARY KEY,
+    username varchar(255) NOT NULL DEFAULT '',
+    token_name varchar(255) NOT NULL DEFAULT '',
+    token varchar(255) NOT NULL DEFAULT '',
+    create_at bigint NOT NULL DEFAULT 0,
+    last_used bigint NOT NULL DEFAULT 0
+);
+
+CREATE TABLE notify_rule (
+    id bigserial PRIMARY KEY,
+    name varchar(255) NOT NULL,
+    description text,
+    enable smallint NOT NULL DEFAULT 0,
+    user_group_ids varchar(255) NOT NULL DEFAULT '',
+    notify_configs text,
+    pipeline_configs text,
+    create_at bigint NOT NULL DEFAULT 0,
+    create_by varchar(64) NOT NULL DEFAULT '',
+    update_at bigint NOT NULL DEFAULT 0,
+    update_by varchar(64) NOT NULL DEFAULT ''
+);
+
+CREATE TABLE notify_channel (
+    id bigserial PRIMARY KEY,
+    name varchar(255) NOT NULL,
+    ident varchar(255) NOT NULL,
+    description text,
+    enable smallint NOT NULL DEFAULT 0,
+    param_config text,
+    request_type varchar(50) NOT NULL,
+    request_config text,
+    weight int NOT NULL DEFAULT 0,
+    create_at bigint NOT NULL DEFAULT 0,
+    create_by varchar(64) NOT NULL DEFAULT '',
+    update_at bigint NOT NULL DEFAULT 0,
+    update_by varchar(64) NOT NULL DEFAULT ''
+);
+
+CREATE TABLE message_template (
+    id bigserial PRIMARY KEY,
+    name varchar(64) NOT NULL,
+    ident varchar(64) NOT NULL,
+    content text,
+    user_group_ids varchar(64),
+    notify_channel_ident varchar(64) NOT NULL DEFAULT '',
+    private int NOT NULL DEFAULT 0,
+    weight int NOT NULL DEFAULT 0,
+    create_at bigint NOT NULL DEFAULT 0,
+    create_by varchar(64) NOT NULL DEFAULT '',
+    update_at bigint NOT NULL DEFAULT 0,
+    update_by varchar(64) NOT NULL DEFAULT ''
+);
+
+CREATE TABLE event_pipeline (
+    id bigserial PRIMARY KEY,
+    name varchar(128) NOT NULL,
+    team_ids text,
+    description varchar(255) NOT NULL DEFAULT '',
+    filter_enable smallint NOT NULL DEFAULT 0,
+    label_filters text,
+    attribute_filters text,
+    processors text,
+    create_at bigint NOT NULL DEFAULT 0,
+    create_by varchar(64) NOT NULL DEFAULT '',
+    update_at bigint NOT NULL DEFAULT 0,
+    update_by varchar(64) NOT NULL DEFAULT ''
+);
+
+CREATE TABLE embedded_product (
+    id bigserial PRIMARY KEY,
+    name varchar(255) DEFAULT NULL,
+    url varchar(255) DEFAULT NULL,
+    is_private boolean DEFAULT NULL,
+    team_ids varchar(255),
+    create_at bigint NOT NULL DEFAULT 0,
+    create_by varchar(64) NOT NULL DEFAULT '',
+    update_at bigint NOT NULL DEFAULT 0,
+    update_by varchar(64) NOT NULL DEFAULT ''
 );

docker/compose-postgres/n9eetc_pg/config.toml

@@ -5,7 +5,7 @@ RunMode = "release"
 # log write dir
 Dir = "logs"
 # log level: DEBUG INFO WARNING ERROR
-Level = "DEBUG"
+Level = "INFO"
 # stdout, stderr, file
 Output = "stdout"
 # # rotate by time
@@ -130,8 +130,6 @@ AlertDetail = true
 [Pushgw]
 # use target labels in database instead of in series
 LabelRewrite = true
-# # default busigroup key name
-# BusiGroupLabelKey = "busigroup"
 ForceUseServerTS = true
 
 # [Pushgw.DebugSample]

docker/compose-postgres/n9eetc_pg/template/dingtalk.tpl

@@ -1,25 +0,0 @@
-#### {{if .IsRecovered}}<font color="#008800">💚{{.RuleName}}</font>{{else}}<font color="#FF0000">💔{{.RuleName}}</font>{{end}}
-
----
-{{$time_duration := sub now.Unix .FirstTriggerTime }}{{if .IsRecovered}}{{$time_duration = sub .LastEvalTime .FirstTriggerTime }}{{end}}
-- **告警级别**: {{.Severity}}级
-{{- if .RuleNote}}
-- **规则备注**: {{.RuleNote}}
-{{- end}}
-{{- if not .IsRecovered}}
-- **当次触发时值**: {{.TriggerValue}}
-- **当次触发时间**: {{timeformat .TriggerTime}}
-- **告警持续时长**: {{humanizeDurationInterface $time_duration}}
-{{- else}}
-{{- if .AnnotationsJSON.recovery_value}}
-- **恢复时值**: {{formatDecimal .AnnotationsJSON.recovery_value 4}}
-{{- end}}
-- **恢复时间**: {{timeformat .LastEvalTime}}
-- **告警持续时长**: {{humanizeDurationInterface $time_duration}}
-{{- end}}
-- **告警事件标签**:
-{{- range $key, $val := .TagsMap}}
-{{- if ne $key "rulename" }}
-  - `{{$key}}`: `{{$val}}`
-{{- end}}
-{{- end}}

docker/compose-postgres/n9eetc_pg/template/email.tpl

@@ -1,224 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-    <meta charset="UTF-8">
-    <meta http-equiv="X-UA-Compatible" content="ie=edge">
-    <title>夜莺告警通知</title>
-    <style type="text/css">
-        .wrapper {
-            background-color: #f8f8f8;
-            padding: 15px;
-            height: 100%;
-        }
-        .main {
-            width: 600px;
-            padding: 30px;
-            margin: 0 auto;
-            background-color: #fff;
-            font-size: 12px;
-            font-family: verdana,'Microsoft YaHei',Consolas,'Deja Vu Sans Mono','Bitstream Vera Sans Mono';
-        }
-        header {
-            border-radius: 2px 2px 0 0;
-        }
-        header .title {
-            font-size: 14px;
-            color: #333333;
-            margin: 0;
-        }
-        header .sub-desc {
-            color: #333;
-            font-size: 14px;
-            margin-top: 6px;
-            margin-bottom: 0;
-        }
-        hr {
-            margin: 20px 0;
-            height: 0;
-            border: none;
-            border-top: 1px solid #e5e5e5;
-        }
-        em {
-            font-weight: 600;
-        }
-        table {
-            margin: 20px 0;
-            width: 100%;
-        }
-
-        table tbody tr{
-            font-weight: 200;
-            font-size: 12px;
-            color: #666;
-            height: 32px;
-        }
-
-        .succ {
-            background-color: green;
-            color: #fff;
-        }
-
-        .fail {
-            background-color: red;
-            color: #fff;
-        }
-
-        .succ th, .succ td, .fail th, .fail td {
-            color: #fff;
-        }
-
-        table tbody tr th {
-            width: 80px;
-            text-align: right;
-        }
-        .text-right {
-            text-align: right;
-        }
-        .body {
-            margin-top: 24px;
-        }
-        .body-text {
-            color: #666666;
-            -webkit-font-smoothing: antialiased;
-        }
-        .body-extra {
-            -webkit-font-smoothing: antialiased;
-        }
-        .body-extra.text-right a {
-            text-decoration: none;
-            color: #333;
-        }
-        .body-extra.text-right a:hover {
-            color: #666;
-        }
-        .button {
-            width: 200px;
-            height: 50px;
-            margin-top: 20px;
-            text-align: center;
-            border-radius: 2px;
-            background: #2D77EE;
-            line-height: 50px;
-            font-size: 20px;
-            color: #FFFFFF;
-            cursor: pointer;
-        }
-        .button:hover {
-            background: rgb(25, 115, 255);
-            border-color: rgb(25, 115, 255);
-            color: #fff;
-        }
-        footer {
-            margin-top: 10px;
-            text-align: right;
-        }
-        .footer-logo {
-            text-align: right;
-        }
-        .footer-logo-image {
-            width: 108px;
-            height: 27px;
-            margin-right: 10px;
-        }
-        .copyright {
-            margin-top: 10px;
-            font-size: 12px;
-            text-align: right;
-            color: #999;
-            -webkit-font-smoothing: antialiased;
-        }
-    </style>
-</head>
-<body>
-<div class="wrapper">
-    <div class="main">
-        <header>
-            <h3 class="title">{{.RuleName}}</h3>
-            <p class="sub-desc"></p>
-        </header>
-
-        <hr>
-
-        <div class="body">
-            <table cellspacing="0" cellpadding="0" border="0">
-                <tbody>
-                {{if .IsRecovered}}
-                <tr class="succ">
-                    <th>级别状态：</th>
-                    <td>S{{.Severity}} Recovered</td>
-                </tr>
-                {{else}}
-                <tr class="fail">
-                    <th>级别状态：</th>
-                    <td>S{{.Severity}} Triggered</td>
-                </tr>
-                {{end}}
-
-                <tr>
-                    <th>策略备注：</th>
-                    <td>{{.RuleNote}}</td>
-                </tr>
-                <tr>
-                    <th>设备备注：</th>
-                    <td>{{.TargetNote}}</td>
-                </tr>
-                {{if not .IsRecovered}}
-                <tr>
-                    <th>触发时值：</th>
-                    <td>{{.TriggerValue}}</td>
-                </tr>
-                {{end}}
-
-                {{if .TargetIdent}}
-                <tr>
-                    <th>监控对象：</th>
-                    <td>{{.TargetIdent}}</td>
-                </tr>
-                {{end}}
-                <tr>
-                    <th>监控指标：</th>
-                    <td>{{.TagsJSON}}</td>
-                </tr>
-
-                {{if .IsRecovered}}
-                <tr>
-                    <th>恢复时间：</th>
-                    <td>{{timeformat .LastEvalTime}}</td>
-                </tr>
-                {{else}}
-                <tr>
-                    <th>触发时间：</th>
-                    <td>
-                        {{timeformat .TriggerTime}}
-                    </td>
-                </tr>
-                {{end}}
-
-                <tr>
-                    <th>发送时间：</th>
-                    <td>
-                        {{timestamp}}
-                    </td>
-                </tr>
-
-                <tr>
-                    <th>PromQL：</th>
-                    <td>
-                        {{.PromQl}}
-                    </td>
-                </tr>
-                </tbody>
-            </table>
-
-            <hr>
-
-            <footer>
-                <div class="copyright" style="font-style: italic">
-                    报警太多？使用 <a href="https://flashcat.cloud/product/flashduty/" target="_blank">FlashDuty</a> 做告警聚合降噪、排班OnCall！
-                </div>
-            </footer>
-        </div>
-    </div>
-</div>
-</body>
-</html>

docker/compose-postgres/n9eetc_pg/template/feishu.tpl

@@ -1,7 +0,0 @@
-级别状态: S{{.Severity}} {{if .IsRecovered}}Recovered{{else}}Triggered{{end}}
-规则名称: {{.RuleName}}{{if .RuleNote}}
-规则备注: {{.RuleNote}}{{end}}
-监控指标: {{.TagsJSON}}
-{{if .IsRecovered}}恢复时间：{{timeformat .LastEvalTime}}{{else}}触发时间: {{timeformat .TriggerTime}}
-触发时值: {{.TriggerValue}}{{end}}
-发送时间: {{timestamp}}

docker/compose-postgres/n9eetc_pg/template/mm.tpl

@@ -1,7 +0,0 @@
-级别状态: S{{.Severity}} {{if .IsRecovered}}Recovered{{else}}Triggered{{end}}
-规则名称: {{.RuleName}}{{if .RuleNote}}
-规则备注: {{.RuleNote}}{{end}}
-监控指标: {{.TagsJSON}}
-{{if .IsRecovered}}恢复时间：{{timeformat .LastEvalTime}}{{else}}触发时间: {{timeformat .TriggerTime}}
-触发时值: {{.TriggerValue}}{{end}}
-发送时间: {{timestamp}}

docker/compose-postgres/n9eetc_pg/template/subject.tpl

@@ -1 +0,0 @@
-{{if .IsRecovered}}Recovered{{else}}Triggered{{end}}: {{.RuleName}} {{.TagsJSON}}

docker/compose-postgres/n9eetc_pg/template/telegram.tpl

@@ -1,7 +0,0 @@
-**级别状态**: {{if .IsRecovered}}<font color="info">S{{.Severity}} Recovered</font>{{else}}<font color="warning">S{{.Severity}} Triggered</font>{{end}}
-**规则标题**: {{.RuleName}}{{if .RuleNote}}
-**规则备注**: {{.RuleNote}}{{end}}
-**监控指标**: {{.TagsJSON}}
-{{if .IsRecovered}}**恢复时间**：{{timeformat .LastEvalTime}}{{else}}**触发时间**: {{timeformat .TriggerTime}}
-**触发时值**: {{.TriggerValue}}{{end}}
-**发送时间**: {{timestamp}}

docker/compose-postgres/n9eetc_pg/template/wecom.tpl

@@ -1,7 +0,0 @@
-**级别状态**: {{if .IsRecovered}}<font color="info">S{{.Severity}} Recovered</font>{{else}}<font color="warning">S{{.Severity}} Triggered</font>{{end}}
-**规则标题**: {{.RuleName}}{{if .RuleNote}}
-**规则备注**: {{.RuleNote}}{{end}}
-**监控指标**: {{.TagsJSON}}
-{{if .IsRecovered}}**恢复时间**：{{timeformat .LastEvalTime}}{{else}}**触发时间**: {{timeformat .TriggerTime}}
-**触发时值**: {{.TriggerValue}}{{end}}
-**发送时间**: {{timestamp}}

docker/initsql/a-n9e.sql

@@ -107,12 +107,6 @@ insert into `role_operation`(role_name, operation) values('Standard', '/help/mig
 insert into `role_operation`(role_name, operation) values('Standard', '/alert-rules-built-in');
 insert into `role_operation`(role_name, operation) values('Standard', '/dashboards-built-in');
 insert into `role_operation`(role_name, operation) values('Standard', '/trace/dependencies');
-
-insert into `role_operation`(role_name, operation) values('Admin', '/help/source');
-insert into `role_operation`(role_name, operation) values('Admin', '/help/sso');
-insert into `role_operation`(role_name, operation) values('Admin', '/help/notification-tpls');
-insert into `role_operation`(role_name, operation) values('Admin', '/help/notification-settings');
-
 insert into `role_operation`(role_name, operation) values('Standard', '/users');
 insert into `role_operation`(role_name, operation) values('Standard', '/user-groups');
 insert into `role_operation`(role_name, operation) values('Standard', '/user-groups/add');
@@ -291,6 +285,8 @@ CREATE TABLE `alert_rule` (
     `append_tags` varchar(255) not null default '' comment 'split by space: service=n9e mod=api',
     `annotations` text not null comment 'annotations',
     `extra_config` text,
+    `notify_rule_ids` varchar(1024) DEFAULT '',
+    `notify_version` int DEFAULT 0,
     `create_at` bigint not null default 0,
     `create_by` varchar(64) not null default '',
     `update_at` bigint not null default 0,
@@ -351,6 +347,8 @@ CREATE TABLE `alert_subscribe` (
     `extra_config` text,
     `redefine_webhooks` tinyint(1) default 0,
     `for_duration` bigint not null default 0,
+    `notify_rule_ids` varchar(1024) DEFAULT '',
+    `notify_version` int DEFAULT 0,
     `create_at` bigint not null default 0,
     `create_by` varchar(64) not null default '',
     `update_at` bigint not null default 0,
@@ -467,6 +465,7 @@ CREATE TABLE `alert_cur_event` (
     `rule_config` text not null comment 'annotations',
     `tags` varchar(1024) not null default '' comment 'merge data_tags rule_tags, split by ,,',
     `original_tags` text comment 'labels key=val,,k2=v2',
+    `notify_rule_ids` text COMMENT 'notify rule ids',
     PRIMARY KEY (`id`),
     KEY (`hash`),
     KEY (`rule_id`),
@@ -509,6 +508,7 @@ CREATE TABLE `alert_his_event` (
     `original_tags` text comment 'labels key=val,,k2=v2',
     `annotations` text not null comment 'annotations',
     `rule_config` text not null comment 'annotations',
+    `notify_rule_ids` text COMMENT 'notify rule ids',
     PRIMARY KEY (`id`),
     INDEX `idx_last_eval_time` (`last_eval_time`),
     KEY (`hash`),
@@ -533,7 +533,7 @@ CREATE TABLE `builtin_components` (
   `updated_by` varchar(191) NOT NULL DEFAULT '' COMMENT '''updater''',
   `disabled` int NOT NULL DEFAULT 0 COMMENT '''is disabled or not''',
   PRIMARY KEY (`id`),
-  UNIQUE KEY `idx_ident` (`ident`)
+  KEY (`ident`)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
 
 CREATE TABLE `builtin_payloads` (
@@ -560,6 +560,7 @@ CREATE TABLE `builtin_payloads` (
 
 CREATE TABLE notification_record (
     `id` BIGINT PRIMARY KEY AUTO_INCREMENT,
+    `notify_rule_id` BIGINT NOT NULL DEFAULT 0,
     `event_id`  bigint NOT NULL COMMENT 'event history id',
     `sub_id`  bigint COMMENT 'subscribed rule id',
     `channel` varchar(255) NOT NULL COMMENT 'notification channel name',
@@ -631,13 +632,16 @@ CREATE TABLE `alerting_engines`
     `datasource_id` bigint not null default 0 comment 'datasource id',
     `engine_cluster` varchar(128) not null default '' comment 'n9e-alert cluster',
     `clock` bigint not null,
-    PRIMARY KEY (`id`)
+    PRIMARY KEY (`id`),
+    INDEX `idx_inst` (`instance`),
+    INDEX `idx_clock` (`clock`)
 ) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;
 
 CREATE TABLE `datasource`
 (
     `id` int unsigned NOT NULL AUTO_INCREMENT,
     `name` varchar(191) not null default '',
+    `identifier` varchar(255) not null default '',
     `description` varchar(255) not null default '',
     `category` varchar(255) not null default '',
     `plugin_id` int unsigned not null default 0,
@@ -694,6 +698,7 @@ CREATE TABLE `es_index_pattern` (
     `allow_hide_system_indices` tinyint(1) not null default 0,
     `fields_format` varchar(4096) not null default '',
     `cross_cluster_enabled` int not null default 0,
+    `note` varchar(1024) not null default '',
     `create_at` bigint default '0',
     `create_by` varchar(64) default '',
     `update_at` bigint default '0',
@@ -702,6 +707,7 @@ CREATE TABLE `es_index_pattern` (
     UNIQUE KEY (`datasource_id`, `name`)
 ) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;
 
+
 CREATE TABLE `builtin_metrics` (
     `id` bigint unsigned NOT NULL AUTO_INCREMENT COMMENT 'unique identifier',
     `collector` varchar(191) NOT NULL COMMENT '''type of collector''',
@@ -718,6 +724,7 @@ CREATE TABLE `builtin_metrics` (
     `uuid` bigint NOT NULL DEFAULT 0 COMMENT '''uuid''',
     PRIMARY KEY (`id`),
     UNIQUE KEY `idx_collector_typ_name` (`lang`,`collector`, `typ`, `name`),
+    INDEX `idx_uuid` (`uuid`),
     INDEX `idx_collector` (`collector`),
     INDEX `idx_typ` (`typ`),
     INDEX `idx_builtinmetric_name` (`name` ASC),
@@ -774,6 +781,84 @@ CREATE TABLE `user_token` (
     PRIMARY KEY (`id`)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
 
+
+CREATE TABLE `notify_rule` (
+    `id` bigint unsigned not null auto_increment,
+    `name` varchar(255) not null,
+    `description` text,
+    `enable` tinyint(1) not null default 0,
+    `user_group_ids` varchar(255) not null default '',
+    `notify_configs` text,
+    `pipeline_configs` text,
+    `create_at` bigint not null default 0,
+    `create_by` varchar(64) not null default '',
+    `update_at` bigint not null default 0,
+    `update_by` varchar(64) not null default '',
+    PRIMARY KEY (`id`)
+) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;
+
+CREATE TABLE `notify_channel` (
+    `id` bigint unsigned not null auto_increment,
+    `name` varchar(255) not null,
+    `ident` varchar(255) not null,
+    `description` text, 
+    `enable` tinyint(1) not null default 0,
+    `param_config` text,
+    `request_type` varchar(50) not null,
+    `request_config` text,
+    `weight` int not null default 0,
+    `create_at` bigint not null default 0,
+    `create_by` varchar(64) not null default '',
+    `update_at` bigint not null default 0,
+    `update_by` varchar(64) not null default '',
+    PRIMARY KEY (`id`)
+) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;
+
+CREATE TABLE `message_template` (
+    `id` bigint unsigned not null auto_increment,
+    `name` varchar(64) not null,
+    `ident` varchar(64) not null,
+    `content` text,
+    `user_group_ids` varchar(64),
+    `notify_channel_ident` varchar(64) not null default '',
+    `private` int not null default 0,
+    `weight` int not null default 0,
+    `create_at` bigint not null default 0,
+    `create_by` varchar(64) not null default '',
+    `update_at` bigint not null default 0,
+    `update_by` varchar(64) not null default '',
+    PRIMARY KEY (`id`)
+) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;
+
+CREATE TABLE `event_pipeline` (
+    `id` bigint unsigned not null auto_increment,
+    `name` varchar(128) not null,
+    `team_ids` text,
+    `description` varchar(255) not null default '',
+    `filter_enable` tinyint(1) not null default 0,
+    `label_filters` text,
+    `attribute_filters` text,
+    `processors` text,
+    `create_at` bigint not null default 0,
+    `create_by` varchar(64) not null default '',
+    `update_at` bigint not null default 0,
+    `update_by` varchar(64) not null default '',
+    PRIMARY KEY (`id`)
+) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;
+
+CREATE TABLE `embedded_product` (
+    `id` bigint unsigned NOT NULL AUTO_INCREMENT,
+    `name` varchar(255) DEFAULT NULL,
+    `url` varchar(255) DEFAULT NULL,
+    `is_private` boolean DEFAULT NULL,
+    `team_ids` varchar(255),
+    `create_at` bigint not null default 0,
+    `create_by` varchar(64) not null default '',
+    `update_at` bigint not null default 0,
+    `update_by` varchar(64) not null default '',
+    PRIMARY KEY (`id`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+
 CREATE TABLE `task_meta`
 (
     `id`          bigint unsigned NOT NULL AUTO_INCREMENT,
@@ -2130,4 +2215,16 @@ CREATE TABLE task_host_99
     UNIQUE KEY `idx_id_host` (`id`, `host`),
     PRIMARY KEY (`ii`)
 ) ENGINE = InnoDB
-  DEFAULT CHARSET = utf8mb4;
+  DEFAULT CHARSET = utf8mb4;
+
+CREATE TABLE `source_token` (
+    `id` bigint unsigned NOT NULL AUTO_INCREMENT,
+    `source_type` varchar(64) NOT NULL DEFAULT '' COMMENT 'source type',
+    `source_id` varchar(255) NOT NULL DEFAULT '' COMMENT 'source identifier',
+    `token` varchar(255) NOT NULL DEFAULT '' COMMENT 'access token',
+    `expire_at` bigint NOT NULL DEFAULT 0 COMMENT 'expire timestamp',
+    `create_at` bigint NOT NULL DEFAULT 0 COMMENT 'create timestamp',
+    `create_by` varchar(64) NOT NULL DEFAULT '' COMMENT 'creator',
+    PRIMARY KEY (`id`),
+    KEY `idx_source_type_id_token` (`source_type`, `source_id`, `token`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

docker/migratesql/migrate.sql

@@ -133,7 +133,7 @@ ALTER TABLE `builtin_components` MODIFY COLUMN `logo` mediumtext COMMENT '''logo
 /* v8.0.0-beta.2 2024-12-26 */
 ALTER TABLE `es_index_pattern` ADD COLUMN `cross_cluster_enabled` int not null default 0;
 
-/* v8.0.0-beta.3 2024-01-03 */
+/* v8.0.0-beta.3 2025-01-03 */
 ALTER TABLE `builtin_components` ADD COLUMN `disabled` INT NOT NULL DEFAULT 0 COMMENT 'is disabled or not';
         
 CREATE TABLE `dash_annotation` (
@@ -162,4 +162,103 @@ CREATE TABLE `user_token` (
     `create_at` bigint NOT NULL DEFAULT 0,
     `last_used` bigint NOT NULL DEFAULT 0,
     PRIMARY KEY (`id`)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+
+/* v8.0.0-beta.7 2025-03-01 */
+CREATE TABLE `notify_rule` (
+    `id` bigint unsigned not null auto_increment,
+    `name` varchar(255) not null,
+    `description` text,
+    `enable` tinyint(1) not null default 0,
+    `user_group_ids` varchar(255) not null default '',
+    `notify_configs` text,
+    `create_at` bigint not null default 0,
+    `create_by` varchar(64) not null default '',
+    `update_at` bigint not null default 0,
+    `update_by` varchar(64) not null default '',
+    PRIMARY KEY (`id`)
+) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;
+
+CREATE TABLE `notify_channel` (
+    `id` bigint unsigned not null auto_increment,
+    `name` varchar(255) not null,
+    `ident` varchar(255) not null,
+    `description` text, 
+    `enable` tinyint(1) not null default 0,
+    `param_config` text,
+    `request_type` varchar(50) not null,
+    `request_config` text,
+    `create_at` bigint not null default 0,
+    `create_by` varchar(64) not null default '',
+    `update_at` bigint not null default 0,
+    `update_by` varchar(64) not null default '',
+    PRIMARY KEY (`id`)
+) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;
+
+CREATE TABLE `message_template` (
+    `id` bigint unsigned not null auto_increment,
+    `name` varchar(64) not null,
+    `ident` varchar(64) not null,
+    `content` text,
+    `user_group_ids` varchar(64),
+    `notify_channel_ident` varchar(64) not null default '',
+    `private` int not null default 0,
+    `create_at` bigint not null default 0,
+    `create_by` varchar(64) not null default '',
+    `update_at` bigint not null default 0,
+    `update_by` varchar(64) not null default '',
+    PRIMARY KEY (`id`)
+) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;
+
+ALTER TABLE `alert_rule` ADD COLUMN `notify_rule_ids` varchar(1024) DEFAULT '';
+ALTER TABLE `alert_rule` ADD COLUMN `notify_version` int DEFAULT 0;
+
+ALTER TABLE `alert_subscribe` ADD COLUMN `notify_rule_ids` varchar(1024) DEFAULT '';
+ALTER TABLE `alert_subscribe` ADD COLUMN `notify_version` int DEFAULT 0;
+
+ALTER TABLE `notification_record` ADD COLUMN `notify_rule_id` BIGINT NOT NULL DEFAULT 0;
+
+
+/* v8.0.0-beta.9 2025-03-17 */
+ALTER TABLE `message_template` ADD COLUMN `weight` int not null default 0;
+ALTER TABLE `notify_channel` ADD COLUMN `weight` int not null default 0;
+
+/* v8.0.0-beta.11 2025-04-10 */
+ALTER TABLE `es_index_pattern` ADD COLUMN `note` varchar(1024) not null default '';
+ALTER TABLE `datasource` ADD COLUMN `identifier` varchar(255) not null default '';
+
+/* v8.0.0-beta.11 2025-05-15 */
+ALTER TABLE `notify_rule` ADD COLUMN `pipeline_configs` text;
+
+CREATE TABLE `event_pipeline` (
+    `id` bigint unsigned not null auto_increment,
+    `name` varchar(128) not null,
+    `team_ids` text,
+    `description` varchar(255) not null default '',
+    `filter_enable` tinyint(1) not null default 0,
+    `label_filters` text,
+    `attribute_filters` text,
+    `processors` text,
+    `create_at` bigint not null default 0,
+    `create_by` varchar(64) not null default '',
+    `update_at` bigint not null default 0,
+    `update_by` varchar(64) not null default '',
+    PRIMARY KEY (`id`)
+) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;
+
+/* v8.0.0-next */
+CREATE TABLE `source_token` (
+    `id` bigint unsigned NOT NULL AUTO_INCREMENT,
+    `source_type` varchar(64) NOT NULL DEFAULT '' COMMENT 'source type',
+    `source_id` varchar(255) NOT NULL DEFAULT '' COMMENT 'source identifier',
+    `token` varchar(255) NOT NULL DEFAULT '' COMMENT 'access token',
+    `expire_at` bigint NOT NULL DEFAULT 0 COMMENT 'expire timestamp',
+    `create_at` bigint NOT NULL DEFAULT 0 COMMENT 'create timestamp',
+    `create_by` varchar(64) NOT NULL DEFAULT '' COMMENT 'creator',
+    PRIMARY KEY (`id`),
+    KEY `idx_source_type_id_token` (`source_type`, `source_id`, `token`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+
+/* v8.0.0-beta.12 2025-06-03 */
+ALTER TABLE `alert_his_event` ADD COLUMN `notify_rule_ids` text COMMENT 'notify rule ids';
+ALTER TABLE `alert_cur_event` ADD COLUMN `notify_rule_ids` text COMMENT 'notify rule ids';

dscache/sync.go

@@ -9,6 +9,8 @@ import (
 	"github.com/ccfos/nightingale/v6/datasource"
 	_ "github.com/ccfos/nightingale/v6/datasource/ck"
 	"github.com/ccfos/nightingale/v6/datasource/es"
+	_ "github.com/ccfos/nightingale/v6/datasource/mysql"
+	_ "github.com/ccfos/nightingale/v6/datasource/postgresql"
 	"github.com/ccfos/nightingale/v6/dskit/tdengine"
 	"github.com/ccfos/nightingale/v6/models"
 	"github.com/ccfos/nightingale/v6/pkg/ctx"
@@ -47,6 +49,7 @@ var PromDefaultDatasourceId int64
 func getDatasourcesFromDBLoop(ctx *ctx.Context, fromAPI bool) {
 	for {
 		if !fromAPI {
+			foundDefaultDatasource := false
 			items, err := models.GetDatasources(ctx)
 			if err != nil {
 				logger.Errorf("get datasource from database fail: %v", err)
@@ -58,6 +61,7 @@ func getDatasourcesFromDBLoop(ctx *ctx.Context, fromAPI bool) {
 			for _, item := range items {
 				if item.PluginType == "prometheus" && item.IsDefault {
 					atomic.StoreInt64(&PromDefaultDatasourceId, item.Id)
+					foundDefaultDatasource = true
 				}
 
 				logger.Debugf("get datasource: %+v", item)
@@ -90,6 +94,12 @@ func getDatasourcesFromDBLoop(ctx *ctx.Context, fromAPI bool) {
 				}
 				dss = append(dss, ds)
 			}
+
+			if !foundDefaultDatasource && atomic.LoadInt64(&PromDefaultDatasourceId) != 0 {
+				logger.Debugf("no default datasource found")
+				atomic.StoreInt64(&PromDefaultDatasourceId, 0)
+			}
+
 			PutDatasources(dss)
 		} else {
 			FromAPIHook()
@@ -183,7 +193,14 @@ func PutDatasources(items []datasource.DatasourceInfo) {
 		ids = append(ids, item.Id)
 
 		// 异步初始化 client 不然数据源同步的会很慢
-		go DsCache.Put(typ, item.Id, ds)
+		go func() {
+			defer func() {
+				if r := recover(); r != nil {
+					logger.Errorf("panic in datasource item: %+v panic:%v", item, r)
+				}
+			}()
+			DsCache.Put(typ, item.Id, ds)
+		}()
 	}
 
 	logger.Debugf("get plugin by type success Ids:%v", ids)

dskit/mysql/mysql.go

@@ -0,0 +1,172 @@
+// @Author: Ciusyan 5/10/24
+
+package mysql
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/dskit/pool"
+	"github.com/ccfos/nightingale/v6/dskit/sqlbase"
+	"github.com/ccfos/nightingale/v6/dskit/types"
+
+	_ "github.com/go-sql-driver/mysql" // MySQL driver
+	"github.com/mitchellh/mapstructure"
+	"gorm.io/driver/mysql"
+	"gorm.io/gorm"
+)
+
+type MySQL struct {
+	Shards []Shard `json:"mysql.shards" mapstructure:"mysql.shards"`
+}
+
+type Shard struct {
+	Addr            string `json:"mysql.addr" mapstructure:"mysql.addr"`
+	DB              string `json:"mysql.db" mapstructure:"mysql.db"`
+	User            string `json:"mysql.user" mapstructure:"mysql.user"`
+	Password        string `json:"mysql.password" mapstructure:"mysql.password"`
+	Timeout         int    `json:"mysql.timeout" mapstructure:"mysql.timeout"`
+	MaxIdleConns    int    `json:"mysql.max_idle_conns" mapstructure:"mysql.max_idle_conns"`
+	MaxOpenConns    int    `json:"mysql.max_open_conns" mapstructure:"mysql.max_open_conns"`
+	ConnMaxLifetime int    `json:"mysql.conn_max_lifetime" mapstructure:"mysql.conn_max_lifetime"`
+	MaxQueryRows    int    `json:"mysql.max_query_rows" mapstructure:"mysql.max_query_rows"`
+}
+
+func NewMySQLWithSettings(ctx context.Context, settings interface{}) (*MySQL, error) {
+	newest := new(MySQL)
+	settingsMap := map[string]interface{}{}
+
+	switch s := settings.(type) {
+	case string:
+		if err := json.Unmarshal([]byte(s), &settingsMap); err != nil {
+			return nil, err
+		}
+	case map[string]interface{}:
+		settingsMap = s
+	default:
+		return nil, errors.New("unsupported settings type")
+	}
+
+	if err := mapstructure.Decode(settingsMap, newest); err != nil {
+		return nil, err
+	}
+
+	return newest, nil
+}
+
+// NewConn establishes a new connection to MySQL
+func (m *MySQL) NewConn(ctx context.Context, database string) (*gorm.DB, error) {
+	if len(m.Shards) == 0 {
+		return nil, errors.New("empty pgsql shards")
+	}
+
+	shard := m.Shards[0]
+
+	if shard.Timeout == 0 {
+		shard.Timeout = 300
+	}
+
+	if shard.MaxIdleConns == 0 {
+		shard.MaxIdleConns = 10
+	}
+
+	if shard.MaxOpenConns == 0 {
+		shard.MaxOpenConns = 100
+	}
+
+	if shard.ConnMaxLifetime == 0 {
+		shard.ConnMaxLifetime = 300
+	}
+
+	if shard.MaxQueryRows == 0 {
+		shard.MaxQueryRows = 100
+	}
+
+	if len(shard.Addr) == 0 {
+		return nil, errors.New("empty addr")
+	}
+
+	if len(shard.Addr) == 0 {
+		return nil, errors.New("empty addr")
+	}
+	var keys []string
+	var err error
+	keys = append(keys, shard.Addr)
+
+	keys = append(keys, shard.Password, shard.User)
+	if len(database) > 0 {
+		keys = append(keys, database)
+	}
+	cachedKey := strings.Join(keys, ":")
+	// cache conn with database
+	conn, ok := pool.PoolClient.Load(cachedKey)
+	if ok {
+		return conn.(*gorm.DB), nil
+	}
+	var db *gorm.DB
+	defer func() {
+		if db != nil && err == nil {
+			pool.PoolClient.Store(cachedKey, db)
+		}
+	}()
+
+	dsn := fmt.Sprintf("%s:%s@tcp(%s)/%s?charset=utf8&parseTime=True", shard.User, shard.Password, shard.Addr, database)
+
+	return sqlbase.NewDB(
+		ctx,
+		mysql.Open(dsn),
+		shard.MaxIdleConns,
+		shard.MaxOpenConns,
+		time.Duration(shard.ConnMaxLifetime)*time.Second,
+	)
+}
+
+func (m *MySQL) ShowDatabases(ctx context.Context) ([]string, error) {
+	db, err := m.NewConn(ctx, "")
+	if err != nil {
+		return nil, err
+	}
+
+	return sqlbase.ShowDatabases(ctx, db, "SHOW DATABASES")
+}
+
+func (m *MySQL) ShowTables(ctx context.Context, database string) ([]string, error) {
+	db, err := m.NewConn(ctx, database)
+	if err != nil {
+		return nil, err
+	}
+
+	return sqlbase.ShowTables(ctx, db, "SHOW TABLES")
+}
+
+func (m *MySQL) DescTable(ctx context.Context, database, table string) ([]*types.ColumnProperty, error) {
+	db, err := m.NewConn(ctx, database)
+	if err != nil {
+		return nil, err
+	}
+
+	query := fmt.Sprintf("DESCRIBE %s", table)
+	return sqlbase.DescTable(ctx, db, query)
+}
+
+func (m *MySQL) SelectRows(ctx context.Context, database, table, query string) ([]map[string]interface{}, error) {
+	db, err := m.NewConn(ctx, database)
+	if err != nil {
+		return nil, err
+	}
+
+	return sqlbase.SelectRows(ctx, db, table, query)
+}
+
+func (m *MySQL) ExecQuery(ctx context.Context, database string, sql string) ([]map[string]interface{}, error) {
+	db, err := m.NewConn(ctx, database)
+	if err != nil {
+		return nil, err
+	}
+
+	return sqlbase.ExecQuery(ctx, db, sql)
+}

dskit/mysql/mysql_test.go

@@ -0,0 +1,129 @@
+// @Author: Ciusyan 5/11/24
+
+package mysql
+
+import (
+	"context"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestNewMySQLWithSettings(t *testing.T) {
+	tests := []struct {
+		name     string
+		settings interface{}
+		wantErr  bool
+	}{
+		{
+			name:     "valid string settings",
+			settings: `{"mysql.addr":"localhost:3306","mysql.user":"root","mysql.password":"root","mysql.maxIdleConns":5,"mysql.maxOpenConns":10,"mysql.connMaxLifetime":30}`,
+			wantErr:  false,
+		},
+		{
+			name:     "invalid settings type",
+			settings: 12345,
+			wantErr:  true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, err := NewMySQLWithSettings(context.Background(), tt.settings)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("NewMySQLWithSettings() error = %v, wantErr %v", err, tt.wantErr)
+			}
+			t.Log(got)
+		})
+	}
+}
+
+func TestNewConn(t *testing.T) {
+	ctx := context.Background()
+	settings := `{"mysql.addr":"localhost:3306","mysql.user":"root","mysql.password":"root","mysql.maxIdleConns":5,"mysql.maxOpenConns":10,"mysql.connMaxLifetime":30}`
+	mysql, err := NewMySQLWithSettings(ctx, settings)
+	require.NoError(t, err)
+
+	tests := []struct {
+		name     string
+		database string
+		wantErr  bool
+	}{
+		{
+			name:     "valid connection",
+			database: "db1",
+			wantErr:  false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			_, err := mysql.NewConn(ctx, tt.database)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("NewConn() error = %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+		})
+	}
+}
+
+func TestShowDatabases(t *testing.T) {
+	ctx := context.Background()
+	settings := `{"mysql.addr":"localhost:3306","mysql.user":"root","mysql.password":"root","mysql.maxIdleConns":5,"mysql.maxOpenConns":10,"mysql.connMaxLifetime":30}`
+	mysql, err := NewMySQLWithSettings(ctx, settings)
+	require.NoError(t, err)
+
+	databases, err := mysql.ShowDatabases(ctx)
+	require.NoError(t, err)
+	t.Log(databases)
+}
+
+func TestShowTables(t *testing.T) {
+	ctx := context.Background()
+	settings := `{"mysql.addr":"localhost:3306","mysql.user":"root","mysql.password":"root","mysql.maxIdleConns":5,"mysql.maxOpenConns":10,"mysql.connMaxLifetime":30}`
+	mysql, err := NewMySQLWithSettings(ctx, settings)
+	require.NoError(t, err)
+
+	tables, err := mysql.ShowTables(ctx, "db1")
+	require.NoError(t, err)
+	t.Log(tables)
+}
+
+func TestDescTable(t *testing.T) {
+	ctx := context.Background()
+	settings := `{"mysql.addr":"localhost:3306","mysql.user":"root","mysql.password":"root","mysql.maxIdleConns":5,"mysql.maxOpenConns":10,"mysql.connMaxLifetime":30}`
+	mysql, err := NewMySQLWithSettings(ctx, settings)
+	require.NoError(t, err)
+
+	descTable, err := mysql.DescTable(ctx, "db1", "students")
+	require.NoError(t, err)
+	for _, desc := range descTable {
+		t.Logf("%+v", *desc)
+	}
+}
+
+func TestExecQuery(t *testing.T) {
+	ctx := context.Background()
+	settings := `{"mysql.addr":"localhost:3306","mysql.user":"root","mysql.password":"root","mysql.maxIdleConns":5,"mysql.maxOpenConns":10,"mysql.connMaxLifetime":30}`
+	mysql, err := NewMySQLWithSettings(ctx, settings)
+	require.NoError(t, err)
+
+	rows, err := mysql.ExecQuery(ctx, "db1", "SELECT * FROM students WHERE id = 10008")
+	require.NoError(t, err)
+	for _, row := range rows {
+		t.Log(row)
+	}
+}
+
+func TestSelectRows(t *testing.T) {
+	ctx := context.Background()
+	settings := `{"mysql.addr":"localhost:3306","mysql.user":"root","mysql.password":"root","mysql.maxIdleConns":5,"mysql.maxOpenConns":10,"mysql.connMaxLifetime":30}`
+	mysql, err := NewMySQLWithSettings(ctx, settings)
+	require.NoError(t, err)
+
+	rows, err := mysql.SelectRows(ctx, "db1", "students", "id > 10008")
+	require.NoError(t, err)
+	for _, row := range rows {
+		t.Log(row)
+	}
+}

dskit/mysql/timeseries.go

@@ -0,0 +1,74 @@
+package mysql
+
+import (
+	"context"
+	"fmt"
+	"strings"
+
+	"github.com/ccfos/nightingale/v6/dskit/sqlbase"
+	"github.com/ccfos/nightingale/v6/dskit/types"
+
+	"gorm.io/gorm"
+)
+
+// Query executes a given SQL query in MySQL and returns the results
+func (m *MySQL) Query(ctx context.Context, query *sqlbase.QueryParam) ([]map[string]interface{}, error) {
+	db, err := m.NewConn(ctx, "")
+	if err != nil {
+		return nil, err
+	}
+
+	err = m.CheckMaxQueryRows(db, ctx, query)
+	if err != nil {
+		return nil, err
+	}
+
+	return sqlbase.Query(ctx, db, query)
+}
+
+// QueryTimeseries executes a time series data query using the given parameters
+func (m *MySQL) QueryTimeseries(ctx context.Context, query *sqlbase.QueryParam) ([]types.MetricValues, error) {
+	db, err := m.NewConn(ctx, "")
+	if err != nil {
+		return nil, err
+	}
+
+	err = m.CheckMaxQueryRows(db, ctx, query)
+	if err != nil {
+		return nil, err
+	}
+
+	return sqlbase.QueryTimeseries(ctx, db, query)
+}
+
+func (m *MySQL) CheckMaxQueryRows(db *gorm.DB, ctx context.Context, query *sqlbase.QueryParam) error {
+	sql := strings.ReplaceAll(query.Sql, ";", "")
+	checkQuery := &sqlbase.QueryParam{
+		Sql: fmt.Sprintf("SELECT COUNT(*) as count FROM (%s) AS subquery;", sql),
+	}
+
+	res, err := sqlbase.Query(ctx, db, checkQuery)
+	if err != nil {
+		return err
+	}
+
+	if len(res) > 0 {
+		if count, exists := res[0]["count"]; exists {
+			v, err := sqlbase.ParseFloat64Value(count)
+			if err != nil {
+				return err
+			}
+
+			maxQueryRows := m.Shards[0].MaxQueryRows
+			if maxQueryRows == 0 {
+				maxQueryRows = 500
+			}
+
+			if v > float64(maxQueryRows) {
+				return fmt.Errorf("query result rows count %d exceeds the maximum limit %d", int(v), maxQueryRows)
+			}
+		}
+	}
+
+	return nil
+}

dskit/mysql/timeseries_test.go

@@ -0,0 +1,62 @@
+// @Author: Ciusyan 5/11/24
+
+package mysql
+
+import (
+	"context"
+	"testing"
+
+	"github.com/ccfos/nightingale/v6/dskit/sqlbase"
+	"github.com/ccfos/nightingale/v6/dskit/types"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestQuery(t *testing.T) {
+	ctx := context.Background()
+	settings := `{"mysql.addr":"localhost:3306","mysql.user":"root","mysql.password":"root","mysql.maxIdleConns":5,"mysql.maxOpenConns":10,"mysql.connMaxLifetime":30}`
+	mysql, err := NewMySQLWithSettings(ctx, settings)
+	require.NoError(t, err)
+
+	param := &sqlbase.QueryParam{
+		Sql: "SELECT * FROM students WHERE id > 10900",
+		Keys: types.Keys{
+			ValueKey:   "",
+			LabelKey:   "",
+			TimeKey:    "",
+			TimeFormat: "",
+		},
+	}
+
+	rows, err := mysql.Query(ctx, param)
+	require.NoError(t, err)
+	for _, row := range rows {
+		t.Log(row)
+	}
+}
+
+func TestQueryTimeseries(t *testing.T) {
+	ctx := context.Background()
+	settings := `{"mysql.addr":"localhost:3306","mysql.user":"root","mysql.password":"root","mysql.maxIdleConns":5,"mysql.maxOpenConns":10,"mysql.connMaxLifetime":30}`
+	mysql, err := NewMySQLWithSettings(ctx, settings)
+	require.NoError(t, err)
+
+	// Prepare a test query parameter
+	param := &sqlbase.QueryParam{
+		Sql: "SELECT id, grade, student_name, a_grade, update_time FROM students WHERE grade > 20000", // Modify SQL query to select specific columns
+		Keys: types.Keys{
+			ValueKey:   "grade a_grade",                 // Set the value key to the column name containing the metric value
+			LabelKey:   "id student_name",               // Set the label key to the column name containing the metric label
+			TimeKey:    "update_time",                   // Set the time key to the column name containing the timestamp
+			TimeFormat: "2006-01-02 15:04:05 +0000 UTC", // Provide the time format according to the timestamp column's format
+		},
+	}
+
+	// Execute the query and retrieve the time series data
+	metricValues, err := mysql.QueryTimeseries(ctx, param)
+	require.NoError(t, err)
+
+	for _, metric := range metricValues {
+		t.Log(metric)
+	}
+}

dskit/pool/pool.go

@@ -0,0 +1,37 @@
+package pool
+
+import (
+	"bytes"
+	"sync"
+	"time"
+
+	gc "github.com/patrickmn/go-cache"
+)
+
+var (
+	PoolClient = new(sync.Map)
+)
+
+var (
+	// default cache instance, do not use this if you want to specify the defaultExpiration
+	DefaultCache = gc.New(time.Hour*24, time.Hour)
+)
+
+var (
+	bytesPool = sync.Pool{
+		New: func() interface{} { return new(bytes.Buffer) },
+	}
+)
+
+func PoolGetBytesBuffer() *bytes.Buffer {
+	buf := bytesPool.Get().(*bytes.Buffer)
+	buf.Reset()
+	return buf
+}
+
+func PoolPutBytesBuffer(buf *bytes.Buffer) {
+	if buf == nil {
+		return
+	}
+	bytesPool.Put(buf)
+}

dskit/postgres/postgres.go

@@ -0,0 +1,207 @@
+// @Author: Ciusyan 5/20/24
+
+package postgres
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+
+	"github.com/ccfos/nightingale/v6/dskit/pool"
+	"github.com/ccfos/nightingale/v6/dskit/sqlbase"
+	"github.com/ccfos/nightingale/v6/dskit/types"
+
+	_ "github.com/lib/pq" // PostgreSQL driver
+	"github.com/mitchellh/mapstructure"
+	"gorm.io/driver/postgres"
+	"gorm.io/gorm"
+)
+
+type PostgreSQL struct {
+	Shard `json:",inline"  mapstructure:",squash"`
+}
+
+type Shard struct {
+	Addr            string `json:"pgsql.addr" mapstructure:"pgsql.addr"`
+	DB              string `json:"pgsql.db" mapstructure:"pgsql.db"`
+	User            string `json:"pgsql.user" mapstructure:"pgsql.user"`
+	Password        string `json:"pgsql.password" mapstructure:"pgsql.password" `
+	Timeout         int    `json:"pgsql.timeout" mapstructure:"pgsql.timeout"`
+	MaxIdleConns    int    `json:"pgsql.max_idle_conns" mapstructure:"pgsql.max_idle_conns"`
+	MaxOpenConns    int    `json:"pgsql.max_open_conns" mapstructure:"pgsql.max_open_conns"`
+	ConnMaxLifetime int    `json:"pgsql.conn_max_lifetime" mapstructure:"pgsql.conn_max_lifetime"`
+	MaxQueryRows    int    `json:"pgsql.max_query_rows" mapstructure:"pgsql.max_query_rows"`
+}
+
+// NewPostgreSQLWithSettings initializes a new PostgreSQL instance with the given settings
+func NewPostgreSQLWithSettings(ctx context.Context, settings interface{}) (*PostgreSQL, error) {
+	newest := new(PostgreSQL)
+	settingsMap := map[string]interface{}{}
+
+	switch s := settings.(type) {
+	case string:
+		if err := json.Unmarshal([]byte(s), &settingsMap); err != nil {
+			return nil, err
+		}
+	case map[string]interface{}:
+		settingsMap = s
+	case *PostgreSQL:
+		return s, nil
+	case PostgreSQL:
+		return &s, nil
+	case Shard:
+		newest.Shard = s
+		return newest, nil
+	case *Shard:
+		newest.Shard = *s
+		return newest, nil
+	default:
+		return nil, errors.New("unsupported settings type")
+	}
+
+	if err := mapstructure.Decode(settingsMap, newest); err != nil {
+		return nil, err
+	}
+
+	return newest, nil
+}
+
+// NewConn establishes a new connection to PostgreSQL
+func (p *PostgreSQL) NewConn(ctx context.Context, database string) (*gorm.DB, error) {
+	if len(p.DB) == 0 && len(database) == 0 {
+		return nil, errors.New("empty pgsql database") // 兼容阿里实时数仓Holgres, 连接时必须指定db名字
+	}
+
+	if p.Shard.Timeout == 0 {
+		p.Shard.Timeout = 60
+	}
+
+	if p.Shard.MaxIdleConns == 0 {
+		p.Shard.MaxIdleConns = 10
+	}
+
+	if p.Shard.MaxOpenConns == 0 {
+		p.Shard.MaxOpenConns = 100
+	}
+
+	if p.Shard.ConnMaxLifetime == 0 {
+		p.Shard.ConnMaxLifetime = 14400
+	}
+
+	if len(p.Shard.Addr) == 0 {
+		return nil, errors.New("empty fe-node addr")
+	}
+	var keys []string
+	var err error
+	keys = append(keys, p.Shard.Addr)
+
+	keys = append(keys, p.Shard.Password, p.Shard.User)
+	if len(database) > 0 {
+		keys = append(keys, database)
+	}
+	cachedKey := strings.Join(keys, ":")
+	// cache conn with database
+	conn, ok := pool.PoolClient.Load(cachedKey)
+	if ok {
+		return conn.(*gorm.DB), nil
+	}
+
+	var db *gorm.DB
+	defer func() {
+		if db != nil && err == nil {
+			pool.PoolClient.Store(cachedKey, db)
+		}
+	}()
+
+	// Simplified connection logic for PostgreSQL
+	dsn := fmt.Sprintf("postgres://%s:%s@%s/%s?sslmode=disable&TimeZone=Asia/Shanghai", p.Shard.User, p.Shard.Password, p.Shard.Addr, database)
+	db, err = sqlbase.NewDB(
+		ctx,
+		postgres.Open(dsn),
+		p.Shard.MaxIdleConns,
+		p.Shard.MaxOpenConns,
+		time.Duration(p.Shard.ConnMaxLifetime)*time.Second,
+	)
+
+	if err != nil {
+		if db != nil {
+			sqlDB, _ := db.DB()
+			if sqlDB != nil {
+				sqlDB.Close()
+			}
+		}
+		return nil, err
+	}
+
+	return db, nil
+}
+
+// ShowDatabases lists all databases in PostgreSQL
+func (p *PostgreSQL) ShowDatabases(ctx context.Context, searchKeyword string) ([]string, error) {
+	db, err := p.NewConn(ctx, "postgres")
+	if err != nil {
+		return nil, err
+	}
+	sql := fmt.Sprintf("SELECT datname FROM pg_database WHERE datistemplate = false AND datname LIKE %s",
+		"'%"+searchKeyword+"%'")
+	return sqlbase.ShowDatabases(ctx, db, sql)
+}
+
+// ShowTables lists all tables in a given database
+func (p *PostgreSQL) ShowTables(ctx context.Context, searchKeyword string) (map[string][]string, error) {
+	db, err := p.NewConn(ctx, p.DB)
+	if err != nil {
+		return nil, err
+	}
+	sql := fmt.Sprintf("SELECT schemaname, tablename FROM pg_tables WHERE schemaname !='information_schema' and schemaname !='pg_catalog'  and  tablename LIKE %s",
+		"'%"+searchKeyword+"%'")
+	rets, err := sqlbase.ExecQuery(ctx, db, sql)
+	if err != nil {
+		return nil, err
+	}
+	tabs := make(map[string][]string, 3)
+	for _, row := range rets {
+		if val, ok := row["schemaname"].(string); ok {
+			tabs[val] = append(tabs[val], row["tablename"].(string))
+		}
+	}
+	return tabs, nil
+}
+
+// DescTable describes the schema of a specified table in PostgreSQL
+// scheme default: public if not specified
+func (p *PostgreSQL) DescTable(ctx context.Context, scheme, table string) ([]*types.ColumnProperty, error) {
+	db, err := p.NewConn(ctx, p.DB)
+	if err != nil {
+		return nil, err
+	}
+	if scheme == "" {
+		scheme = "public"
+	}
+
+	query := fmt.Sprintf("SELECT column_name, data_type, is_nullable, column_default FROM information_schema.columns WHERE table_name = '%s' AND table_schema = '%s'", table, scheme)
+	return sqlbase.DescTable(ctx, db, query)
+}
+
+// SelectRows selects rows from a specified table in PostgreSQL based on a given query
+func (p *PostgreSQL) SelectRows(ctx context.Context, table, where string) ([]map[string]interface{}, error) {
+	db, err := p.NewConn(ctx, p.DB)
+	if err != nil {
+		return nil, err
+	}
+
+	return sqlbase.SelectRows(ctx, db, table, where)
+}
+
+// ExecQuery executes a SQL query in PostgreSQL
+func (p *PostgreSQL) ExecQuery(ctx context.Context, sql string) ([]map[string]interface{}, error) {
+	db, err := p.NewConn(ctx, p.DB)
+	if err != nil {
+		return nil, err
+	}
+
+	return sqlbase.ExecQuery(ctx, db, sql)
+}

dskit/postgres/timeseries.go

@@ -0,0 +1,73 @@
+package postgres
+
+import (
+	"context"
+	"fmt"
+	"strings"
+
+	"github.com/ccfos/nightingale/v6/dskit/sqlbase"
+	"github.com/ccfos/nightingale/v6/dskit/types"
+	"gorm.io/gorm"
+)
+
+// Query executes a given SQL query in PostgreSQL and returns the results
+func (p *PostgreSQL) Query(ctx context.Context, query *sqlbase.QueryParam) ([]map[string]interface{}, error) {
+	db, err := p.NewConn(ctx, p.Shard.DB)
+	if err != nil {
+		return nil, err
+	}
+
+	err = p.CheckMaxQueryRows(db, ctx, query)
+	if err != nil {
+		return nil, err
+	}
+
+	return sqlbase.Query(ctx, db, query)
+}
+
+// QueryTimeseries executes a time series data query using the given parameters
+func (p *PostgreSQL) QueryTimeseries(ctx context.Context, query *sqlbase.QueryParam) ([]types.MetricValues, error) {
+	db, err := p.NewConn(ctx, p.Shard.DB)
+	if err != nil {
+		return nil, err
+	}
+
+	err = p.CheckMaxQueryRows(db, ctx, query)
+	if err != nil {
+		return nil, err
+	}
+
+	return sqlbase.QueryTimeseries(ctx, db, query, true)
+}
+
+func (p *PostgreSQL) CheckMaxQueryRows(db *gorm.DB, ctx context.Context, query *sqlbase.QueryParam) error {
+	sql := strings.ReplaceAll(query.Sql, ";", "")
+	checkQuery := &sqlbase.QueryParam{
+		Sql: fmt.Sprintf("SELECT COUNT(*) as count FROM (%s) AS subquery;", sql),
+	}
+
+	res, err := sqlbase.Query(ctx, db, checkQuery)
+	if err != nil {
+		return err
+	}
+
+	if len(res) > 0 {
+		if count, exists := res[0]["count"]; exists {
+			v, err := sqlbase.ParseFloat64Value(count)
+			if err != nil {
+				return err
+			}
+
+			maxQueryRows := p.Shard.MaxQueryRows
+			if maxQueryRows == 0 {
+				maxQueryRows = 500
+			}
+
+			if v > float64(maxQueryRows) {
+				return fmt.Errorf("query result rows count %d exceeds the maximum limit %d", int(v), maxQueryRows)
+			}
+		}
+	}
+
+	return nil
+}

dskit/sqlbase/base.go

@@ -9,9 +9,9 @@ import (
 	"strings"
 	"time"
 
-	"github.com/ccfos/nightingale/v6/dskit/types"
-
 	"gorm.io/gorm"
+
+	"github.com/ccfos/nightingale/v6/dskit/types"
 )
 
 // NewDB creates a new Gorm DB instance based on the provided gorm.Dialector and configures the connection pool
@@ -19,7 +19,7 @@ func NewDB(ctx context.Context, dialector gorm.Dialector, maxIdleConns, maxOpenC
 	// Create a new Gorm DB instance
 	db, err := gorm.Open(dialector, &gorm.Config{})
 	if err != nil {
-		return nil, err
+		return db, err
 	}
 
 	// Configure the connection pool
@@ -35,6 +35,17 @@ func NewDB(ctx context.Context, dialector gorm.Dialector, maxIdleConns, maxOpenC
 	return db.WithContext(ctx), sqlDB.Ping()
 }
 
+func CloseDB(db *gorm.DB) error {
+	if db != nil {
+		sqlDb, err := db.DB()
+		if err != nil {
+			return err
+		}
+		return sqlDb.Close()
+	}
+	return nil
+}
+
 // ShowTables retrieves a list of all tables in the specified database
 func ShowTables(ctx context.Context, db *gorm.DB, query string) ([]string, error) {
 	var tables []string
@@ -112,7 +123,7 @@ func DescTable(ctx context.Context, db *gorm.DB, query string) ([]*types.ColumnP
 		}
 
 		// Convert the database-specific type to internal type
-		type2, indexable := convertDBType(db.Dialector.Name(), typ)
+		type2, indexable := ConvertDBType(db.Dialector.Name(), typ)
 		columns = append(columns, &types.ColumnProperty{
 			Field:     field,
 			Type:      typ,
@@ -175,7 +186,7 @@ func SelectRows(ctx context.Context, db *gorm.DB, table, query string) ([]map[st
 }
 
 // convertDBType converts MySQL or PostgreSQL data types to custom internal types and determines if they are indexable
-func convertDBType(dialect, dbType string) (string, bool) {
+func ConvertDBType(dialect, dbType string) (string, bool) {
 	typ := strings.ToLower(dbType)
 
 	// Common type conversions
@@ -190,7 +201,7 @@ func convertDBType(dialect, dbType string) (string, bool) {
 		strings.HasPrefix(typ, "char"), strings.HasPrefix(typ, "tinytext"),
 		strings.HasPrefix(typ, "mediumtext"), strings.HasPrefix(typ, "longtext"),
 		strings.HasPrefix(typ, "character varying"), strings.HasPrefix(typ, "nvarchar"),
-		strings.HasPrefix(typ, "nchar"):
+		strings.HasPrefix(typ, "nchar"), strings.HasPrefix(typ, "bpchar"):
 		return types.LogExtractValueTypeText, true
 
 	case strings.HasPrefix(typ, "float"), strings.HasPrefix(typ, "double"),
@@ -203,7 +214,7 @@ func convertDBType(dialect, dbType string) (string, bool) {
 		strings.HasPrefix(typ, "time"), strings.HasPrefix(typ, "smalldatetime"):
 		return types.LogExtractValueTypeDate, false
 
-	case strings.HasPrefix(typ, "boolean"), strings.HasPrefix(typ, "bit"):
+	case strings.HasPrefix(typ, "boolean"), strings.HasPrefix(typ, "bit"), strings.HasPrefix(typ, "bool"):
 		return types.LogExtractValueTypeBool, false
 	}