TIP

研发工作中遇到过最为抓狂的事情可能就是排查线上问题了。明显易发现的问题还好,但有些问题只在生产环境的高并发真实流量场景中才会出现,相同的分支代码放到线下又可以正常运行了,每当此时也只能以问题很难复现再观察定位的话术来敷衍。那么生产环境问题为什么难定位呢:

    🟢 生产环境不能debug,看不到流程的全貌,很多时候流程真正的执行链路需要靠猜测

    🟢 虽然可以通过日志查看到很多细节和问题,但日志又不能无限制的使用,不然即影响了开发效率,又影响了代码的可读性,日志加的太多还容易造成生产环境机器的计算资源、存储资源浪费

    🟢 一些极端场景的链路或数据,只有生产环境才会出现

    为了减少问题排查的复杂程度,Kstry 提供了流程回溯功能,流程回溯记录了节点的信息、执行顺序、耗时、入参、出参等重要数据

# 6.1 traceId支持

Kstry 内部使用MDC工具,支持设置全链路请求ID

    🟢 StoryEngine.fire入参中有requestId属性可以指定当前请求的唯一ID,如果未指定引擎会使用UUID生成一个

    🟢 日志打印时可以获取 traceId 来串联当前请求的全部日志,traceId 默认名是:ks-request-id

<property name="FILE_LOG_PATTERN" value="%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread][%X{ks-request-id}] %-5level %logger{20} - %msg%n"/>
1

    🟢 traceId 默认名可以修改:

# application.yml
kstry:
  story:
    request-id: traceId
1
2
3
4

# 6.2 流程回溯

# 6.2.1 流程回溯类型

Kstry 支持五种模式的回溯

    🟢 NONE:不开启回溯功能

    🟢 ALL:开启全部采集点的回溯,包括所有bpmn元素执行,节点信息,节点入参,节点出参,耗时等

    🟢 NODE:只采集bpmn节点执行流程,不关注节点执行细节

    🟢 SERVICE:只关注bpmn元素中的 ServiceTask 节点,也就是服务节点的执行,采集节点的信息、耗时、执行顺序等,但是不采集出参、入参

    🟢 SERVICE_DETAIL:在SERVICE的基础上增加了对出参、入参的采集

# 6.2.2 流程回溯使用

执行入参中显示指定:

@RestController
@RequestMapping("/goods")
public class GoodsController {

    @Resource
    private StoryEngine storyEngine;

    @PostMapping("/show")
    public GoodsDetail showGoods(@RequestBody GoodsDetailRequest request) {

        StoryRequest<GoodsDetail> req = ReqBuilder.returnType(GoodsDetail.class).startId("kstry-demo-goods-show").trackingType(TrackingTypeEnum.SERVICE).request(request).build();
        TaskResponse<GoodsDetail> fire = storyEngine.fire(req);
        if (fire.isSuccess()) {
            return fire.getResult();
        }
        return null;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

全局默认配置:

# application.yml

kstry:
  story:
    tracking:
      type: service
      log: true # 是否打印回溯log,默认为 false
1
2
3
4
5
6
7

# 6.2.3 回溯日志样例

    🟢 ALL

[K1040009] startId: kstry-demo-goods-show, spend 631ms:  [{
    "index": 1,
    "nodeId": "kstry-demo-goods-show",
    "nodeName": "kstry-demo-goods-show",
    "nodeType": "START_EVENT",
    "noticeTracking": [],
    "paramTracking": [],
    "startTime": "2021-12-17T01:36:20.926",
    "threadId": "kstry-task-thread-pool-0",
    "toNodeIds": ["Flow_1uuw3lx"]
}, {
    "index": 2,
    "nodeId": "Flow_1uuw3lx",
    "nodeType": "SEQUENCE_FLOW",
    "noticeTracking": [],
    "paramTracking": [],
    "startTime": "2021-12-17T01:36:20.926",
    "threadId": "kstry-task-thread-pool-0",
    "toNodeIds": ["Activity_1t7q7ro"]
}, {
    "endTime": "2021-12-17T01:36:20.993",
    "index": 3,
    "methodName": "initBaseInfo",
    "nodeId": "Activity_1t7q7ro",
    "nodeName": "初始化\n基本信息",
    "nodeType": "SERVICE_TASK",
    "noticeTracking": [{
        "fieldName": "goodsDetail",
        "noticeName": "goodsDetail",
        "noticeScopeType": "STABLE",
        "value": "{\"id\":1,\"img\":\"https://xxx.png\",\"name\":\"商品\",\"needEvaluate\":true}"
    }, {
        "noticeScopeType": "RESULT",
        "value": "{\"id\":1,\"img\":\"https://xxx.png\",\"name\":\"商品\",\"needEvaluate\":true}"
    }],
    "paramTracking": [{
        "paramName": "request",
        "sourceName": "request",
        "sourceScopeType": "REQUEST",
        "value": "{\"businessId\":\"special-channel\",\"id\":1,\"source\":\"app\"}"
    }, {
        "paramName": "goodsDetail",
        "sourceName": "goodsDetail",
        "sourceScopeType": "STABLE",
        "value": "{\"id\":1,\"img\":\"https://xxx.png\",\"name\":\"商品\",\"needEvaluate\":true}"
    }, {
        "paramName": "role",
        "sourceName": "role",
        "sourceScopeType": "EMPTY",
        "value": "null"
    }],
    "spendTime": 67,
    "startTime": "2021-12-17T01:36:20.926",
    "targetName": "cn.kstry.demo.service.GoodsService",
    "threadId": "kstry-task-thread-pool-0",
    "toNodeIds": ["Flow_0dt6q7w"]
}
......
{
    "endTime": "2021-12-17T01:36:21.549",
    "index": 46,
    "methodName": "detailPostProcess",
    "nodeId": "Activity_1s51gmf",
    "nodeName": "商详后置处理",
    "nodeType": "SERVICE_TASK",
    "noticeTracking": [],
    "paramTracking": [{
        "paramName": "request.shopInfo",
        "sourceName": "shopInfo",
        "sourceScopeType": "STABLE",
        "value": "{\"id\":1,\"labels\":[{\"index\":1,\"name\":\"L1\",\"url\":\"https://xxx.png\"},{\"index\":2,\"name\":\"L2\",\"url\":\"https://xxx2.png\"}],\"salesNumMonthly\":22,\"shopName\":\"店铺名称\"}"
    }, {
        "paramName": "request.goodsDetail",
        "sourceName": "goodsDetail",
        "sourceScopeType": "STABLE",
        "value": "{\"id\":1,\"img\":\"https://xxx.png\",\"name\":\"商品\",\"needEvaluate\":true,\"skuInfos\":[{\"img\":\"https://xxx.png\",\"price\":1000,\"skuId\":100,\"skuName\":\"sku1\",\"stock\":10},{\"img\":\"https://xxx2.png\",\"price\":2000,\"skuId\":101,\"skuName\":\"sku2\",\"stock\":20}]}"
    }, {
        "paramName": "request.logisticInsurance",
        "sourceName": "logisticInsurance",
        "sourceScopeType": "STABLE",
        "value": "{\"desc\":\"运费险描述\",\"type\":1}"
    }, {
        "paramName": "request.evaluationInfo",
        "sourceName": "evaluationInfo",
        "sourceScopeType": "STABLE",
        "value": "{\"evaluateCount\":20}"
    }, {
        "paramName": "request.orderInfo",
        "sourceName": "orderInfo",
        "sourceScopeType": "STABLE",
        "value": "{\"orderedCount\":10}"
    }, {
        "paramName": "request.goodsExtInfo",
        "sourceName": "goodsExtInfo",
        "sourceScopeType": "STABLE",
        "value": "{\"collectCount\":30}"
    }],
    "spendTime": 11,
    "startTime": "2021-12-17T01:36:21.538",
    "targetName": "cn.kstry.demo.service.GoodsService",
    "threadId": "kstry-task-thread-pool-8",
    "toNodeIds": ["Flow_0rl59u8"]
}, {
    "index": 47,
    "nodeId": "Flow_0rl59u8",
    "nodeType": "SEQUENCE_FLOW",
    "noticeTracking": [],
    "paramTracking": [],
    "startTime": "2021-12-17T01:36:21.549",
    "threadId": "kstry-task-thread-pool-8",
    "toNodeIds": ["Event_1yjag2c"]
}, {
    "index": 48,
    "nodeId": "Event_1yjag2c",
    "nodeType": "END_EVENT",
    "noticeTracking": [],
    "paramTracking": [],
    "startTime": "2021-12-17T01:36:21.550",
    "threadId": "kstry-task-thread-pool-8",
    "toNodeIds": []
}]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121

    🟢 NODE

[K1040009] startId: kstry-demo-goods-show, spend 772ms: 
[{
    "index": 1,
    "nodeId": "kstry-demo-goods-show",
    "nodeName": "kstry-demo-goods-show",
    "nodeType": "START_EVENT",
    "noticeTracking": [],
    "paramTracking": [],
    "toNodeIds": ["Flow_1uuw3lx"]
}, {
    "index": 2,
    "nodeId": "Flow_1uuw3lx",
    "nodeType": "SEQUENCE_FLOW",
    "noticeTracking": [],
    "paramTracking": [],
    "toNodeIds": ["Activity_1t7q7ro"]
}, {
    "index": 3,
    "nodeId": "Activity_1t7q7ro",
    "nodeName": "初始化\n基本信息",
    "nodeType": "SERVICE_TASK",
    "noticeTracking": [],
    "paramTracking": [],
    "toNodeIds": ["Flow_0dt6q7w"]
}
......
{
    "index": 46,
    "nodeId": "Activity_1s51gmf",
    "nodeName": "商详后置处理",
    "nodeType": "SERVICE_TASK",
    "noticeTracking": [],
    "paramTracking": [],
    "toNodeIds": ["Flow_0rl59u8"]
}, {
    "index": 47,
    "nodeId": "Flow_0rl59u8",
    "nodeType": "SEQUENCE_FLOW",
    "noticeTracking": [],
    "paramTracking": [],
    "toNodeIds": ["Event_1yjag2c"]
}, {
    "index": 48,
    "nodeId": "Event_1yjag2c",
    "nodeType": "END_EVENT",
    "noticeTracking": [],
    "paramTracking": [],
    "toNodeIds": []
}]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

    🟢 SERVICE

[K1040009] startId: kstry-demo-goods-show, spend 325ms: 
[{
    "endTime": "2021-12-17T01:41:54.152",
    "index": 3,
    "methodName": "initBaseInfo",
    "nodeId": "Activity_1t7q7ro",
    "nodeName": "初始化\n基本信息",
    "nodeType": "SERVICE_TASK",
    "noticeTracking": [],
    "paramTracking": [],
    "spendTime": 57,
    "startTime": "2021-12-17T01:41:54.095",
    "targetName": "cn.kstry.demo.service.GoodsService",
    "threadId": "kstry-task-thread-pool-0",
    "toNodeIds": ["Activity_1iu4r2e"]
}, {
    "ability": "triple",
    "endTime": "2021-12-17T01:41:54.190",
    "index": 5,
    "methodName": "tripleCheckImg",
    "nodeId": "Activity_1iu4r2e",
    "nodeName": "风控服务",
    "nodeType": "SERVICE_TASK",
    "noticeTracking": [],
    "paramTracking": [],
    "spendTime": 33,
    "startTime": "2021-12-17T01:41:54.157",
    "targetName": "cn.kstry.demo.service.RiskControlService",
    "threadId": "kstry-task-thread-pool-0",
    "toNodeIds": ["Activity_1nhha2f"]
}
......
{
    "endTime": "2021-12-17T01:41:54.243",
    "index": 42,
    "methodName": "getShopInfoByGoodsId",
    "nodeId": "Activity_019xur8",
    "nodeName": "加载店铺信息",
    "nodeType": "SERVICE_TASK",
    "noticeTracking": [],
    "paramTracking": [],
    "spendTime": 13,
    "startTime": "2021-12-17T01:41:54.230",
    "targetName": "cn.kstry.demo.service.ShopService",
    "threadId": "kstry-task-thread-pool-2",
    "toNodeIds": ["Activity_1s51gmf"]
}, {
    "endTime": "2021-12-17T01:41:54.409",
    "index": 46,
    "methodName": "detailPostProcess",
    "nodeId": "Activity_1s51gmf",
    "nodeName": "商详后置处理",
    "nodeType": "SERVICE_TASK",
    "noticeTracking": [],
    "paramTracking": [],
    "spendTime": 10,
    "startTime": "2021-12-17T01:41:54.399",
    "targetName": "cn.kstry.demo.service.GoodsService",
    "threadId": "kstry-task-thread-pool-8",
    "toNodeIds": []
}]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

    🟢 SERVICE_DETAIL

 [K1040009] startId: kstry-demo-goods-show, spend 284ms:
 [{
     "endTime": "2021-12-17T01:43:43.177",
     "index": 3,
     "methodName": "initBaseInfo",
     "nodeId": "Activity_1t7q7ro",
     "nodeName": "初始化\n基本信息",
     "nodeType": "SERVICE_TASK",
     "noticeTracking": [{
         "fieldName": "goodsDetail",
         "noticeName": "goodsDetail",
         "noticeScopeType": "STABLE",
         "value": "{\"id\":1,\"img\":\"https://xxx.png\",\"name\":\"商品\",\"needEvaluate\":true}"
     }, {
         "noticeScopeType": "RESULT",
         "value": "{\"id\":1,\"img\":\"https://xxx.png\",\"name\":\"商品\",\"needEvaluate\":true}"
     }],
     "paramTracking": [{
         "paramName": "request",
         "sourceName": "request",
         "sourceScopeType": "REQUEST",
         "value": "{\"businessId\":\"special-channel\",\"id\":1,\"source\":\"app\"}"
     }, {
         "paramName": "goodsDetail",
         "sourceName": "goodsDetail",
         "sourceScopeType": "STABLE",
         "value": "{\"id\":1,\"img\":\"https://xxx.png\",\"name\":\"商品\",\"needEvaluate\":true}"
     }, {
         "paramName": "role",
         "sourceName": "role",
         "sourceScopeType": "EMPTY",
         "value": "null"
     }],
     "spendTime": 57,
     "startTime": "2021-12-17T01:43:43.120",
     "targetName": "cn.kstry.demo.service.GoodsService",
     "threadId": "kstry-task-thread-pool-0",
     "toNodeIds": ["Activity_1iu4r2e"]
 }, {
     "ability": "triple",
     "endTime": "2021-12-17T01:43:43.210",
     "index": 5,
     "methodName": "tripleCheckImg",
     "nodeId": "Activity_1iu4r2e",
     "nodeName": "风控服务",
     "nodeType": "SERVICE_TASK",
     "noticeTracking": [],
     "paramTracking": [{
         "paramName": "checkInfo.img",
         "sourceName": "goodsDetail.img",
         "sourceScopeType": "STABLE",
         "value": "https://xxx.png"
     }],
     "spendTime": 29,
     "startTime": "2021-12-17T01:43:43.181",
     "targetName": "cn.kstry.demo.service.RiskControlService",
     "threadId": "kstry-task-thread-pool-0",
     "toNodeIds": ["Activity_1nhha2f"]
 }
 ...... 
 {
     "endTime": "2021-12-17T01:43:43.260",
     "index": 42,
     "methodName": "getShopInfoByGoodsId",
     "nodeId": "Activity_019xur8",
     "nodeName": "加载店铺信息",
     "nodeType": "SERVICE_TASK",
     "noticeTracking": [{
         "fieldName": "shopInfo",
         "noticeName": "shopInfo",
         "noticeScopeType": "STABLE",
         "value": "{\"id\":1,\"salesNumMonthly\":22,\"shopName\":\"店铺名称\"}"
     }],
     "paramTracking": [{
         "paramName": "goodsId",
         "sourceName": "id",
         "sourceScopeType": "REQUEST",
         "value": "1"
     }],
     "spendTime": 13,
     "startTime": "2021-12-17T01:43:43.247",
     "targetName": "cn.kstry.demo.service.ShopService",
     "threadId": "kstry-task-thread-pool-2",
     "toNodeIds": ["Activity_1s51gmf"]
 }, {
     "endTime": "2021-12-17T01:43:43.394",
     "index": 46,
     "methodName": "detailPostProcess",
     "nodeId": "Activity_1s51gmf",
     "nodeName": "商详后置处理",
     "nodeType": "SERVICE_TASK",
     "noticeTracking": [],
     "paramTracking": [{
         "paramName": "request.shopInfo",
         "sourceName": "shopInfo",
         "sourceScopeType": "STABLE",
         "value": "{\"id\":1,\"labels\":[{\"index\":1,\"name\":\"L1\",\"url\":\"https://xxx.png\"},{\"index\":2,\"name\":\"L2\",\"url\":\"https://xxx2.png\"}],\"salesNumMonthly\":22,\"shopName\":\"店铺名称\"}"
     }, {
         "paramName": "request.goodsDetail",
         "sourceName": "goodsDetail",
         "sourceScopeType": "STABLE",
         "value": "{\"id\":1,\"img\":\"https://xxx.png\",\"name\":\"商品\",\"needEvaluate\":true,\"skuInfos\":[{\"img\":\"https://xxx.png\",\"price\":1000,\"skuId\":100,\"skuName\":\"sku1\",\"stock\":10},{\"img\":\"https://xxx2.png\",\"price\":2000,\"skuId\":101,\"skuName\":\"sku2\",\"stock\":20}]}"
     }, {
         "paramName": "request.logisticInsurance",
         "sourceName": "logisticInsurance",
         "sourceScopeType": "STABLE",
         "value": "{\"desc\":\"运费险描述\",\"type\":1}"
     }, {
         "paramName": "request.evaluationInfo",
         "sourceName": "evaluationInfo",
         "sourceScopeType": "STABLE",
         "value": "{\"evaluateCount\":20}"
     }, {
         "paramName": "request.orderInfo",
         "sourceName": "orderInfo",
         "sourceScopeType": "STABLE",
         "value": "{\"orderedCount\":10}"
     }, {
         "paramName": "request.goodsExtInfo",
         "sourceName": "goodsExtInfo",
         "sourceScopeType": "STABLE",
         "value": "{\"collectCount\":30}"
     }],
     "spendTime": 15,
     "startTime": "2021-12-17T01:43:43.379",
     "targetName": "cn.kstry.demo.service.GoodsService",
     "threadId": "kstry-task-thread-pool-8",
     "toNodeIds": []
 }]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

# 6.3 节点状态事件

TIP

框架提供了TrackingBeginEventTrackingFinishEvent两个Spring容器事件,用来监听节点执行前后的日志对象,可以通过订阅Spring事件的方式使用

@Component
public class TrackingEventListener {

    @EventListener
    public void handleTrackingBeginEvent(TrackingBeginEvent event) {
        System.out.println("TrackingBeginEvent:" + JSON.toJSONString(event));
    }

    @EventListener
    public void handleTrackingFinishEvent(TrackingFinishEvent event) {
        System.out.println("TrackingFinishEvent:" + JSON.toJSONString(event));
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14

    🟢 TrackingBeginEvent:节点开始执行前触发

    🟢 TrackingFinishEvent:节点执行完成后触发

# 6.4 回溯自定义

回溯自定义是在链路执行完之后,拿到结果或者异常之前,执行的自定义回调方法,可以应对如下问题:

    🟢 自定义流程回溯日志,甚至是出现异常时才打印回溯日志

    🟢 检查节点执行,参数设置等是否符合预期。有时结果没有报错,并不代表一定是没有问题的

    🟢 如果链路中有自定义角色的操作,检查最终角色是否符合预期

举例:

按时间维度的先后执行顺序,打印节点名及耗时:

@RestController
@RequestMapping("/goods")
public class GoodsController {

    @Resource
    private StoryEngine storyEngine;

    @PostMapping("/show")
    public GoodsDetail showGoods(@RequestBody GoodsDetailRequest request) {

        StoryRequest<GoodsDetail> req = ReqBuilder.returnType(GoodsDetail.class).startId("kstry-demo-goods-show")
                .trackingType(TrackingTypeEnum.SERVICE_DETAIL).request(request).build();
      
        req.setRecallStoryHook(recallStory -> {
            MonitorTracking monitorTracking = recallStory.getMonitorTracking();
            List<NodeTracking> storyTracking = monitorTracking.getStoryTracking();
            List<String> collect = storyTracking.stream()
                    .map(nt -> GlobalUtil.format("{}({})", nt.getNodeName(), nt.getSpendTime())).collect(Collectors.toList());
            System.out.println("name list: " + String.join(",", collect));
        });
        TaskResponse<GoodsDetail> fire = storyEngine.fire(req);
        if (fire.isSuccess()) {
            return fire.getResult();
        }
        return null;
    }
}

// 日志: name list: 初始化\n基本信息(73),风控服务(33),三方服务统计(1),加载SKU信息(8),加载广告(null),送运费险(30),加载收藏数(6),加载下单数(6),加载评价数(2),加载店铺信息(19),商详后置处理(15)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29