Building

Started building at 2021/12/10 09:40:12
Calculating base
Updating mirror
Basing run on 7.1.0-1886 a97303989
Updating tree for run 10.12.2021-09.40
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at b908442, changes since last good build: none
kv_engine is at 77fe9b6, changes since last good build: none
ns_server is at e02bf21, changes since last good build: 
 e02bf2141 Remove using mochijson2 encoding in event logs
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 6975b4e, changes since last good build: none
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at de9f5f4, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable a6fcfc03] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable e7102218] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 53b14695] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 19360d9f] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable edc35331] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable 93556d41] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable 8308d170] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable 54f8de19] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable cb4a435] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/10 09:56:27
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/10 09:57:55 Started functional
2021/12/10 09:58:04 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/10 09:58:09 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/10 09:58:09 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 09:58:15 Removing node(s): [] from the cluster
2021/12/10 09:58:20 Rebalance progress: 0
2021/12/10 09:58:25 Rebalance progress: 0
2021/12/10 09:58:30 Rebalance progress: 4.62962962962963
2021/12/10 09:58:35 Rebalance progress: 9.25925925925926
2021/12/10 09:58:40 Rebalance progress: 12.73148148148148
2021/12/10 09:58:45 Rebalance progress: 17.39728009259259
2021/12/10 09:58:50 Rebalance progress: 20.83333333333333
2021/12/10 09:58:55 Rebalance progress: 25.46296296296296
2021/12/10 09:59:00 Rebalance progress: 100
2021/12/10 09:59:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 09:59:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 09:59:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 09:59:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 10:00:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 10:00:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 10:00:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 10:00:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 10:01:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 10:01:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 10:01:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/10 10:01:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/10 10:02:05 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:02:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/10 10:02:12 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 10:02:17 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 10:02:22 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 10:02:27 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 10:02:27 App: TestAdvancedGetOps got deployed
2021/12/10 10:02:27 Starting bucket ops to source bucket
2021/12/10 10:02:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 10:02:37 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 10:02:37 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 10:02:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:02:37 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:02:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:02:42 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 10:02:47 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 10:02:52 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 10:02:57 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 10:02:57 App: TestAdvancedGetOps got un-deployed
2021/12/10 10:02:57 No eventing-con process running
2021/12/10 10:02:57 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/10 10:02:59 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:03:09 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/10 10:03:14 Waiting for dst bucket item count to get to: 0 curr count: 46
2021/12/10 10:03:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:03:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:03:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:03:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (80.04s)
=== RUN   TestAdvancedInsertOps
2021/12/10 10:03:25 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:03:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 10:03:32 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 10:03:37 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 10:03:42 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 10:03:47 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 10:03:47 App: TestAdvancedInsertOps got deployed
2021/12/10 10:03:47 Starting bucket ops to source bucket
2021/12/10 10:03:53 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 10:03:58 Waiting for dst bucket item count to get to: 1024 curr count: 845
2021/12/10 10:04:03 Waiting for dst bucket item count to get to: 1024 curr count: 845
2021/12/10 10:04:03 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/10 10:04:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:04:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:04:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:04:08 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 10:04:13 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 10:04:18 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 10:04:23 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 10:04:23 App: TestAdvancedInsertOps got un-deployed
2021/12/10 10:04:23 No eventing-con process running
2021/12/10 10:04:23 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/10 10:04:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:04:34 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 10:04:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:04:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:04:46 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 10:04:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:04:46 Testing insert operation on source bucket
2021/12/10 10:04:46 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:04:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 10:04:52 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 10:04:57 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 10:05:02 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 10:05:07 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 10:05:07 App: TestAdvancedInsertOps got deployed
2021/12/10 10:05:07 Starting bucket ops to source bucket
2021/12/10 10:05:13 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/10 10:05:18 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/10 10:05:23 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/10 10:05:23 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/10 10:05:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:05:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:05:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:05:28 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 10:05:33 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 10:05:38 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 10:05:43 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 10:05:43 App: TestAdvancedInsertOps got un-deployed
2021/12/10 10:05:43 No eventing-con process running
2021/12/10 10:05:44 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/10 10:05:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:05:55 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/10 10:05:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:05:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:06:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.52s)
=== RUN   TestAdvancedUpsertOps
2021/12/10 10:06:02 Starting bucket ops to source bucket
2021/12/10 10:06:03 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:06:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 10:06:09 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 10:06:14 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 10:06:19 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 10:06:24 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 10:06:24 App: TestAdvancedUpsertOps got deployed
2021/12/10 10:06:24 Starting bucket ops to source bucket
2021/12/10 10:06:30 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 10:06:35 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/10 10:06:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:06:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:06:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:06:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:06:40 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 10:06:45 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 10:06:50 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 10:06:55 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 10:06:55 App: TestAdvancedUpsertOps got un-deployed
2021/12/10 10:06:55 No eventing-con process running
2021/12/10 10:06:55 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/10 10:06:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:07:07 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 10:07:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:07:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:07:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:07:13 Testing upsert operation on source bucket
2021/12/10 10:07:13 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:07:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 10:07:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 10:07:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 10:07:30 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 10:07:35 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 10:07:35 App: TestAdvancedUpsertOps got deployed
2021/12/10 10:07:35 Starting bucket ops to source bucket
2021/12/10 10:07:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:07:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:07:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:07:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:07:41 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 10:07:46 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 10:07:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 10:07:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 10:07:56 App: TestAdvancedUpsertOps got un-deployed
2021/12/10 10:07:56 No eventing-con process running
2021/12/10 10:07:56 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/10 10:07:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:08:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:08:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:08:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (127.86s)
=== RUN   TestAdvancedReplaceOps
2021/12/10 10:08:10 Starting bucket ops to source bucket
2021/12/10 10:08:11 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:08:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/10 10:08:17 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 10:08:22 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 10:08:27 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 10:08:32 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 10:08:32 App: TestAdvancedReplaceOps got deployed
2021/12/10 10:08:32 Starting bucket ops to source bucket
2021/12/10 10:08:38 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 10:08:43 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 10:08:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:08:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:08:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:08:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:08:48 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 10:08:53 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 10:08:58 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 10:09:03 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 10:09:03 App: TestAdvancedReplaceOps got un-deployed
2021/12/10 10:09:03 No eventing-con process running
2021/12/10 10:09:03 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/10 10:09:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:09:15 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 10:09:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:09:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:09:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:09:22 Testing upsert operation on source bucket
2021/12/10 10:09:22 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:09:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/10 10:09:28 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 10:09:33 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 10:09:38 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 10:09:43 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 10:09:43 App: TestAdvancedReplaceOps got deployed
2021/12/10 10:09:43 Starting bucket ops to source bucket
2021/12/10 10:09:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:09:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:09:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:09:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:09:49 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 10:09:54 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 10:09:59 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 10:10:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 10:10:04 App: TestAdvancedReplaceOps got un-deployed
2021/12/10 10:10:04 No eventing-con process running
2021/12/10 10:10:04 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/10 10:10:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:10:16 Waiting for dst bucket item count to get to: 0 curr count: 380
2021/12/10 10:10:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:10:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:10:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (132.68s)
=== RUN   TestAdvancedDeleteOps
2021/12/10 10:10:22 Starting bucket ops to source bucket
2021/12/10 10:10:23 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:10:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/10 10:10:29 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 10:10:34 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 10:10:39 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 10:10:44 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 10:10:44 App: TestAdvancedDeleteOps got deployed
2021/12/10 10:10:44 Starting bucket ops to source bucket
2021/12/10 10:10:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:10:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:10:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:10:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:10:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:10:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:10:59 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 10:11:04 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 10:11:09 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 10:11:14 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 10:11:14 App: TestAdvancedDeleteOps got un-deployed
2021/12/10 10:11:14 No eventing-con process running
2021/12/10 10:11:14 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/10 10:11:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:11:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:11:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:11:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:11:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:11:33 Testing delete operation on source bucket
2021/12/10 10:11:33 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:11:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/10 10:11:39 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 10:11:44 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 10:11:49 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 10:11:54 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 10:11:54 App: TestAdvancedDeleteOps got deployed
2021/12/10 10:11:54 Starting bucket ops to source bucket
2021/12/10 10:11:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:11:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:11:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:11:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:11:59 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 10:12:04 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 10:12:09 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 10:12:14 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 10:12:14 App: TestAdvancedDeleteOps got un-deployed
2021/12/10 10:12:14 No eventing-con process running
2021/12/10 10:12:14 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/10 10:12:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:12:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:12:23 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:12:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.64s)
=== RUN   TestEnoentAdvancedGet
2021/12/10 10:12:28 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:12:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/10 10:12:34 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 10:12:39 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 10:12:44 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 10:12:49 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 10:12:49 App: TestEnoentAdvancedGet got deployed
2021/12/10 10:12:49 Starting bucket ops to source bucket
2021/12/10 10:12:54 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 10:12:59 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 10:13:04 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 10:13:04 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 10:13:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:13:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:13:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:13:10 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 10:13:15 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 10:13:20 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 10:13:25 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 10:13:25 App: TestEnoentAdvancedGet got un-deployed
2021/12/10 10:13:25 No eventing-con process running
2021/12/10 10:13:25 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/10 10:13:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:13:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:13:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:13:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:13:48 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/10 10:13:53 Waiting for dst bucket item count to get to: 0 curr count: 24
2021/12/10 10:13:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:13:53 Testing get enoent operation on source bucket
2021/12/10 10:13:53 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:13:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/10 10:13:59 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 10:14:04 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 10:14:09 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 10:14:14 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 10:14:14 App: TestEnoentAdvancedGet got deployed
2021/12/10 10:14:14 Starting bucket ops to source bucket
2021/12/10 10:14:19 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/10 10:14:24 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 10:14:24 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/10 10:14:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:14:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:14:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:14:30 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 10:14:35 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 10:14:40 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 10:14:45 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 10:14:45 App: TestEnoentAdvancedGet got un-deployed
2021/12/10 10:14:45 No eventing-con process running
2021/12/10 10:14:45 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/10 10:14:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:14:56 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 10:14:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:14:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:15:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.94s)
=== RUN   TestEnoentAdvancedReplace
2021/12/10 10:15:03 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:15:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 10:15:09 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 10:15:14 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 10:15:19 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 10:15:24 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 10:15:24 App: TestEnoentAdvancedReplace got deployed
2021/12/10 10:15:24 Starting bucket ops to source bucket
2021/12/10 10:15:29 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 10:15:34 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 10:15:34 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 10:15:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:15:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:15:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:15:39 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 10:15:44 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 10:15:49 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 10:15:54 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 10:15:54 App: TestEnoentAdvancedReplace got un-deployed
2021/12/10 10:15:54 No eventing-con process running
2021/12/10 10:15:54 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/10 10:15:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:16:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:16:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:16:18 Waiting for dst bucket item count to get to: 0 curr count: 67
2021/12/10 10:16:23 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/10 10:16:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:16:23 Testing get enoent operation on source bucket
2021/12/10 10:16:23 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:16:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 10:16:29 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 10:16:34 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 10:16:39 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 10:16:44 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 10:16:44 App: TestEnoentAdvancedReplace got deployed
2021/12/10 10:16:44 Starting bucket ops to source bucket
2021/12/10 10:16:49 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/10 10:16:54 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 10:16:54 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/10 10:16:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:16:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:16:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:16:59 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 10:17:04 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 10:17:09 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 10:17:15 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 10:17:15 App: TestEnoentAdvancedReplace got un-deployed
2021/12/10 10:17:15 No eventing-con process running
2021/12/10 10:17:15 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/10 10:17:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:17:26 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 10:17:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:17:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:17:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (149.95s)
=== RUN   TestEnoentAdvancedDelete
2021/12/10 10:17:33 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:17:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 10:17:39 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 10:17:44 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 10:17:49 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 10:17:54 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 10:17:54 App: TestEnoentAdvancedDelete got deployed
2021/12/10 10:17:54 Starting bucket ops to source bucket
2021/12/10 10:17:59 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 10:18:04 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 10:18:04 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 10:18:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:18:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:18:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:18:09 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 10:18:14 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 10:18:19 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 10:18:24 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 10:18:24 App: TestEnoentAdvancedDelete got un-deployed
2021/12/10 10:18:24 No eventing-con process running
2021/12/10 10:18:24 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/10 10:18:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:18:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:18:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:18:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:18:48 Waiting for dst bucket item count to get to: 0 curr count: 68
2021/12/10 10:18:53 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/10 10:18:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:18:53 Testing get enoent operation on source bucket
2021/12/10 10:18:53 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:18:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 10:18:59 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 10:19:04 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 10:19:09 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 10:19:14 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 10:19:14 App: TestEnoentAdvancedDelete got deployed
2021/12/10 10:19:14 Starting bucket ops to source bucket
2021/12/10 10:19:19 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/10 10:19:24 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 10:19:24 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/10 10:19:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:19:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:19:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:19:29 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 10:19:34 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 10:19:39 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 10:19:44 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 10:19:44 App: TestEnoentAdvancedDelete got un-deployed
2021/12/10 10:19:44 No eventing-con process running
2021/12/10 10:19:44 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/10 10:19:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:19:56 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 10:19:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:19:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:20:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.90s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/10 10:20:03 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:20:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/10 10:20:09 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 10:20:14 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 10:20:19 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 10:20:24 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 10:20:24 App: TestAdvancedInsertKeyExist got deployed
2021/12/10 10:20:24 Starting bucket ops to source bucket
2021/12/10 10:20:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:20:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:20:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:20:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:20:29 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 10:20:34 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 10:20:39 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 10:20:44 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 10:20:44 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/10 10:20:44 No eventing-con process running
2021/12/10 10:20:44 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/10 10:20:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:20:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:20:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:20:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:20:58 Pumping document in the destination bucket
2021/12/10 10:20:58 Starting bucket ops to source bucket
2021/12/10 10:20:59 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:21:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/10 10:21:05 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 10:21:10 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 10:21:15 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 10:21:20 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 10:21:20 App: TestAdvancedInsertKeyExist got deployed
2021/12/10 10:21:20 Starting bucket ops to source bucket
2021/12/10 10:21:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:21:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:21:35 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:21:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:21:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:21:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:21:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:21:40 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 10:21:45 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 10:21:50 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 10:21:55 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 10:21:55 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/10 10:21:55 No eventing-con process running
2021/12/10 10:21:55 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/10 10:21:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:22:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:22:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:22:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:22:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.58s)
=== RUN   TestExipryGet
2021/12/10 10:22:14 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:22:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/10 10:22:20 Waiting for app: TestExipryGet to get deployed
2021/12/10 10:22:25 Waiting for app: TestExipryGet to get deployed
2021/12/10 10:22:30 Waiting for app: TestExipryGet to get deployed
2021/12/10 10:22:35 Waiting for app: TestExipryGet to get deployed
2021/12/10 10:22:35 App: TestExipryGet got deployed
2021/12/10 10:22:35 Starting bucket ops to source bucket
2021/12/10 10:23:20 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:23:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:23:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:23:20 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:23:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:23:25 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 10:23:30 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 10:23:35 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 10:23:40 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 10:23:40 App: TestExipryGet got un-deployed
2021/12/10 10:23:40 No eventing-con process running
2021/12/10 10:23:40 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/10 10:23:42 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:23:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:23:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:23:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:23:54 Pumping document in the destination bucket for expiry get
2021/12/10 10:23:54 Starting bucket ops to source bucket
2021/12/10 10:23:55 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:23:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/10 10:24:01 Waiting for app: TestExipryGet to get deployed
2021/12/10 10:24:06 Waiting for app: TestExipryGet to get deployed
2021/12/10 10:24:11 Waiting for app: TestExipryGet to get deployed
2021/12/10 10:24:16 Waiting for app: TestExipryGet to get deployed
2021/12/10 10:24:16 App: TestExipryGet got deployed
2021/12/10 10:24:16 Starting bucket ops to source bucket
2021/12/10 10:25:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:25:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:25:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:25:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:25:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:25:06 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 10:25:11 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 10:25:16 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 10:25:21 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 10:25:21 App: TestExipryGet got un-deployed
2021/12/10 10:25:21 No eventing-con process running
2021/12/10 10:25:21 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/10 10:25:23 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:25:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:25:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:25:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:25:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:25:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (211.05s)
=== RUN   TestExipryInsert
2021/12/10 10:25:45 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:25:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/10 10:25:51 Waiting for app: TestExipryInsert to get deployed
2021/12/10 10:25:56 Waiting for app: TestExipryInsert to get deployed
2021/12/10 10:26:01 Waiting for app: TestExipryInsert to get deployed
2021/12/10 10:26:06 Waiting for app: TestExipryInsert to get deployed
2021/12/10 10:26:06 App: TestExipryInsert got deployed
2021/12/10 10:26:06 Starting bucket ops to source bucket
2021/12/10 10:26:51 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 10:26:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:26:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:26:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:26:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:26:56 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 10:27:01 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 10:27:06 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 10:27:11 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 10:27:11 App: TestExipryInsert got un-deployed
2021/12/10 10:27:11 No eventing-con process running
2021/12/10 10:27:11 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/10 10:27:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:27:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:27:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:27:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:27:25 Testing on the destination bucket for expiry insert
2021/12/10 10:27:25 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:27:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/10 10:27:31 Waiting for app: TestExipryInsert to get deployed
2021/12/10 10:27:36 Waiting for app: TestExipryInsert to get deployed
2021/12/10 10:27:41 Waiting for app: TestExipryInsert to get deployed
2021/12/10 10:27:46 Waiting for app: TestExipryInsert to get deployed
2021/12/10 10:27:46 App: TestExipryInsert got deployed
2021/12/10 10:27:46 Starting bucket ops to source bucket
2021/12/10 10:28:31 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:28:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:28:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:28:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:28:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:28:37 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 10:28:42 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 10:28:47 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 10:28:52 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 10:28:52 App: TestExipryInsert got un-deployed
2021/12/10 10:28:52 No eventing-con process running
2021/12/10 10:28:52 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/10 10:28:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:29:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:29:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:29:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:29:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:29:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.51s)
=== RUN   TestExipryReplace
2021/12/10 10:29:15 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:29:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 10:29:21 Waiting for app: TestExipryReplace to get deployed
2021/12/10 10:29:26 Waiting for app: TestExipryReplace to get deployed
2021/12/10 10:29:31 Waiting for app: TestExipryReplace to get deployed
2021/12/10 10:29:36 Waiting for app: TestExipryReplace to get deployed
2021/12/10 10:29:36 App: TestExipryReplace got deployed
2021/12/10 10:29:36 Starting bucket ops to source bucket
2021/12/10 10:30:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:30:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:30:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:30:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:30:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:30:27 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 10:30:32 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 10:30:37 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 10:30:42 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 10:30:42 App: TestExipryReplace got un-deployed
2021/12/10 10:30:42 No eventing-con process running
2021/12/10 10:30:42 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/10 10:30:43 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:30:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:30:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:30:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:30:56 Testing on the destination bucket for expiry insert
2021/12/10 10:30:56 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:30:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 10:31:02 Waiting for app: TestExipryReplace to get deployed
2021/12/10 10:31:07 Waiting for app: TestExipryReplace to get deployed
2021/12/10 10:31:12 Waiting for app: TestExipryReplace to get deployed
2021/12/10 10:31:17 Waiting for app: TestExipryReplace to get deployed
2021/12/10 10:31:17 App: TestExipryReplace got deployed
2021/12/10 10:31:17 Starting bucket ops to source bucket
2021/12/10 10:32:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:32:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:32:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:32:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:32:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:32:07 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 10:32:12 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 10:32:17 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 10:32:22 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 10:32:22 App: TestExipryReplace got un-deployed
2021/12/10 10:32:22 No eventing-con process running
2021/12/10 10:32:22 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/10 10:32:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:32:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:32:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:32:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:32:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:32:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.57s)
=== RUN   TestExipryUpsert
2021/12/10 10:32:46 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:32:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/10 10:32:52 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 10:32:57 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 10:33:02 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 10:33:07 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 10:33:07 App: TestExipryUpsert got deployed
2021/12/10 10:33:07 Starting bucket ops to source bucket
2021/12/10 10:33:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:33:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:33:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:33:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:33:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:33:57 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 10:34:02 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 10:34:07 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 10:34:12 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 10:34:12 App: TestExipryUpsert got un-deployed
2021/12/10 10:34:12 No eventing-con process running
2021/12/10 10:34:12 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/10 10:34:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:34:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:34:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:34:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:34:26 Testing on the destination bucket for expiry insert
2021/12/10 10:34:26 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:34:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/10 10:34:32 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 10:34:37 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 10:34:42 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 10:34:47 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 10:34:47 App: TestExipryUpsert got deployed
2021/12/10 10:34:47 Starting bucket ops to source bucket
2021/12/10 10:35:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:35:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:35:43 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:35:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:35:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:35:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:35:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:35:48 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 10:35:53 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 10:35:58 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 10:36:03 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 10:36:03 App: TestExipryUpsert got un-deployed
2021/12/10 10:36:03 No eventing-con process running
2021/12/10 10:36:03 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/10 10:36:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:36:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:36:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:36:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:36:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (215.55s)
=== RUN   TestCasReplace
2021/12/10 10:36:21 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:36:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 10:36:27 Waiting for app: TestCasReplace to get deployed
2021/12/10 10:36:32 Waiting for app: TestCasReplace to get deployed
2021/12/10 10:36:37 Waiting for app: TestCasReplace to get deployed
2021/12/10 10:36:42 Waiting for app: TestCasReplace to get deployed
2021/12/10 10:36:43 App: TestCasReplace got deployed
2021/12/10 10:36:43 Starting bucket ops to source bucket
2021/12/10 10:36:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:36:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:36:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:36:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:36:48 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 10:36:53 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 10:36:58 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 10:37:03 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 10:37:03 App: TestCasReplace got un-deployed
2021/12/10 10:37:03 No eventing-con process running
2021/12/10 10:37:03 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/10 10:37:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:37:14 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/10 10:37:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:37:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:37:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:37:21 Testing on the destination bucket for upsert with cas
2021/12/10 10:37:21 Starting bucket ops to source bucket
2021/12/10 10:37:21 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:37:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 10:37:27 Waiting for app: TestCasReplace to get deployed
2021/12/10 10:37:32 Waiting for app: TestCasReplace to get deployed
2021/12/10 10:37:37 Waiting for app: TestCasReplace to get deployed
2021/12/10 10:37:42 Waiting for app: TestCasReplace to get deployed
2021/12/10 10:37:42 App: TestCasReplace got deployed
2021/12/10 10:37:42 Starting bucket ops to source bucket
2021/12/10 10:37:48 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:37:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:37:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:37:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:37:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:37:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:37:58 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 10:38:03 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 10:38:08 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 10:38:13 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 10:38:13 App: TestCasReplace got un-deployed
2021/12/10 10:38:13 No eventing-con process running
2021/12/10 10:38:13 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/10 10:38:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:38:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:38:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:38:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:38:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.04s)
=== RUN   TestCasDelete
2021/12/10 10:38:31 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:38:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 10:38:38 Waiting for app: TestCasDelete to get deployed
2021/12/10 10:38:43 Waiting for app: TestCasDelete to get deployed
2021/12/10 10:38:48 Waiting for app: TestCasDelete to get deployed
2021/12/10 10:38:53 Waiting for app: TestCasDelete to get deployed
2021/12/10 10:38:53 App: TestCasDelete got deployed
2021/12/10 10:38:53 Starting bucket ops to source bucket
2021/12/10 10:38:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:38:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:38:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:38:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:38:58 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 10:39:03 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 10:39:08 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 10:39:13 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 10:39:13 App: TestCasDelete got un-deployed
2021/12/10 10:39:13 No eventing-con process running
2021/12/10 10:39:13 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/10 10:39:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:39:25 Waiting for dst bucket item count to get to: 0 curr count: 11
2021/12/10 10:39:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:39:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:39:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:39:31 Testing on the destination bucket for delete with cas
2021/12/10 10:39:31 Starting bucket ops to source bucket
2021/12/10 10:39:32 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:39:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 10:39:38 Waiting for app: TestCasDelete to get deployed
2021/12/10 10:39:43 Waiting for app: TestCasDelete to get deployed
2021/12/10 10:39:48 Waiting for app: TestCasDelete to get deployed
2021/12/10 10:39:53 Waiting for app: TestCasDelete to get deployed
2021/12/10 10:39:53 App: TestCasDelete got deployed
2021/12/10 10:39:53 Starting bucket ops to source bucket
2021/12/10 10:39:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:40:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:40:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:40:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:40:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 10:40:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:40:08 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 10:40:13 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 10:40:18 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 10:40:23 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 10:40:23 App: TestCasDelete got un-deployed
2021/12/10 10:40:23 No eventing-con process running
2021/12/10 10:40:23 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/10 10:40:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:40:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:40:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:40:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:40:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.27s)
=== RUN   TestCountersIncrement
2021/12/10 10:40:41 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 10:40:47 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 10:40:47 Removing node(s): [] from the cluster
2021/12/10 10:40:52 Rebalance progress: 100
2021/12/10 10:40:52 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:40:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/10 10:40:59 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 10:41:04 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 10:41:09 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 10:41:14 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 10:41:14 App: TestCountersIncrement got deployed
2021/12/10 10:41:14 Starting bucket ops to source bucket
2021/12/10 10:41:20 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 10:41:25 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 10:41:25 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/10 10:41:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:41:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:41:30 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 10:41:35 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 10:41:40 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 10:41:45 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 10:41:45 App: TestCountersIncrement got un-deployed
2021/12/10 10:41:45 No eventing-con process running
2021/12/10 10:41:45 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/10 10:41:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:41:57 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/10 10:41:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:41:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:42:08 Waiting for dst bucket item count to get to: 0 curr count: 869
2021/12/10 10:42:13 Waiting for dst bucket item count to get to: 0 curr count: 357
2021/12/10 10:42:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:42:13 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:42:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/10 10:42:20 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 10:42:25 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 10:42:30 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 10:42:35 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 10:42:35 App: TestCountersIncrement got deployed
2021/12/10 10:42:35 Starting bucket ops to source bucket
2021/12/10 10:42:41 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/10 10:42:46 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/10 10:42:46 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/10 10:42:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:42:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:42:51 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 10:42:56 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 10:43:01 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 10:43:06 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 10:43:06 App: TestCountersIncrement got un-deployed
2021/12/10 10:43:06 No eventing-con process running
2021/12/10 10:43:06 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/10 10:43:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:43:18 Waiting for dst bucket item count to get to: 0 curr count: 1049
2021/12/10 10:43:23 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/10 10:43:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:43:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:43:34 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/10 10:43:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:43:34 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/10 10:43:40 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.05s)
=== RUN   TestCountersDecrement
2021/12/10 10:43:40 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 10:43:40 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 10:43:40 Removing node(s): [] from the cluster
2021/12/10 10:43:45 Rebalance progress: 100
2021/12/10 10:43:45 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:43:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/10 10:43:51 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 10:43:56 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 10:44:01 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 10:44:06 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 10:44:06 App: TestCountersDecrement got deployed
2021/12/10 10:44:06 Starting bucket ops to source bucket
2021/12/10 10:44:12 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 10:44:17 Waiting for dst bucket item count to get to: 1024 curr count: 947
2021/12/10 10:44:22 Waiting for dst bucket item count to get to: 1024 curr count: 947
2021/12/10 10:44:22 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/10 10:44:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:44:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:44:28 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 10:44:33 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 10:44:38 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 10:44:43 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 10:44:43 App: TestCountersDecrement got un-deployed
2021/12/10 10:44:43 No eventing-con process running
2021/12/10 10:44:43 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/10 10:44:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:44:54 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/10 10:44:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:44:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:45:06 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 10:45:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:45:06 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:45:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/10 10:45:12 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 10:45:17 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 10:45:22 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 10:45:27 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 10:45:27 App: TestCountersDecrement got deployed
2021/12/10 10:45:27 Starting bucket ops to source bucket
2021/12/10 10:45:33 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/10 10:45:38 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/10 10:45:43 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/10 10:45:43 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/10 10:45:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:45:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:45:49 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 10:45:54 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 10:45:59 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 10:46:04 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 10:46:04 App: TestCountersDecrement got un-deployed
2021/12/10 10:46:04 No eventing-con process running
2021/12/10 10:46:04 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/10 10:46:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:46:15 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/10 10:46:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:46:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:46:27 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/10 10:46:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:46:27 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/10 10:46:32 Rebalance progress: 100
--- PASS: TestCountersDecrement (172.73s)
=== RUN   TestCacheGet
2021/12/10 10:46:32 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:46:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/10 10:46:38 Waiting for app: TestCacheGet to get deployed
2021/12/10 10:46:43 Waiting for app: TestCacheGet to get deployed
2021/12/10 10:46:48 Waiting for app: TestCacheGet to get deployed
2021/12/10 10:46:53 Waiting for app: TestCacheGet to get deployed
2021/12/10 10:46:54 App: TestCacheGet got deployed
2021/12/10 10:46:54 Starting bucket ops to source bucket
2021/12/10 10:46:59 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/10 10:47:04 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/10 10:47:09 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/10 10:47:14 Waiting for dst bucket item count to get to: 20 curr count: 8
2021/12/10 10:47:19 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/10 10:47:24 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/10 10:47:24 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/10 10:47:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:47:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:47:29 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 10:47:34 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 10:47:39 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 10:47:44 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 10:47:44 App: TestCacheGet got un-deployed
2021/12/10 10:47:44 No eventing-con process running
2021/12/10 10:47:44 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/10 10:47:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:47:56 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/10 10:47:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:47:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:48:07 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/10 10:48:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.17s)
=== RUN   TestCacheRYOW
2021/12/10 10:48:07 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:48:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/10 10:48:14 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 10:48:19 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 10:48:24 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 10:48:29 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 10:48:29 App: TestCacheRYOW got deployed
2021/12/10 10:48:29 Starting bucket ops to source bucket
2021/12/10 10:48:34 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/10 10:48:39 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/10 10:48:39 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/10 10:48:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:48:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:48:44 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 10:48:49 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 10:48:54 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 10:48:59 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 10:48:59 App: TestCacheRYOW got un-deployed
2021/12/10 10:48:59 No eventing-con process running
2021/12/10 10:48:59 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/10 10:49:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:49:11 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:49:16 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:49:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:49:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:49:28 Waiting for dst bucket item count to get to: 0 curr count: 29
2021/12/10 10:49:33 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/10 10:49:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (85.34s)
=== RUN   TestCacheExpiry
2021/12/10 10:49:33 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:49:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/10 10:49:39 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 10:49:44 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 10:49:49 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 10:49:54 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 10:49:54 App: TestCacheExpiry got deployed
2021/12/10 10:49:54 Starting bucket ops to source bucket
2021/12/10 10:49:59 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 10:50:04 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/10 10:50:09 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/10 10:50:14 Waiting for dst bucket item count to get to: 100 curr count: 72
2021/12/10 10:50:19 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/10 10:50:19 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 10:50:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:50:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:50:24 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 10:50:29 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 10:50:34 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 10:50:39 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 10:50:39 App: TestCacheExpiry got un-deployed
2021/12/10 10:50:39 No eventing-con process running
2021/12/10 10:50:39 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/10 10:50:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:50:51 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:50:56 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:50:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:50:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:51:08 Waiting for dst bucket item count to get to: 0 curr count: 66
2021/12/10 10:51:13 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/10 10:51:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.16s)
=== RUN   TestCacheExpiryCustom
2021/12/10 10:51:13 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:51:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/10 10:51:19 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 10:51:24 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 10:51:29 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 10:51:34 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 10:51:34 App: TestCacheExpiryCustom got deployed
2021/12/10 10:51:34 Starting bucket ops to source bucket
2021/12/10 10:51:39 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 10:51:44 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/10 10:51:49 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/10 10:51:54 Waiting for dst bucket item count to get to: 100 curr count: 72
2021/12/10 10:51:59 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/10 10:51:59 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 10:51:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:51:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:52:04 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 10:52:09 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 10:52:15 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 10:52:20 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 10:52:20 App: TestCacheExpiryCustom got un-deployed
2021/12/10 10:52:20 No eventing-con process running
2021/12/10 10:52:20 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/10 10:52:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:52:31 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:52:36 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:52:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:52:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:52:48 Waiting for dst bucket item count to get to: 0 curr count: 70
2021/12/10 10:52:53 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/10 10:52:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.13s)
=== RUN   TestCacheOverflowCustom
2021/12/10 10:52:53 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:52:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/10 10:52:59 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 10:53:04 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 10:53:09 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 10:53:14 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 10:53:14 App: TestCacheOverflowCustom got deployed
2021/12/10 10:53:14 Starting bucket ops to source bucket
2021/12/10 10:53:19 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/10 10:53:24 Waiting for dst bucket item count to get to: 35 curr count: 7
2021/12/10 10:53:29 Waiting for dst bucket item count to get to: 35 curr count: 16
2021/12/10 10:53:34 Waiting for dst bucket item count to get to: 35 curr count: 23
2021/12/10 10:53:39 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/10 10:53:44 Waiting for dst bucket item count to get to: 35 curr count: 30
2021/12/10 10:53:44 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/10 10:53:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:53:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:53:50 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 10:53:55 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 10:54:00 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 10:54:05 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 10:54:05 App: TestCacheOverflowCustom got un-deployed
2021/12/10 10:54:05 No eventing-con process running
2021/12/10 10:54:05 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/10 10:54:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:54:16 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/10 10:54:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:54:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:54:28 Waiting for dst bucket item count to get to: 0 curr count: 27
2021/12/10 10:54:33 Waiting for dst bucket item count to get to: 0 curr count: 9
2021/12/10 10:54:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.29s)
=== RUN   TestStrictMode
2021/12/10 10:54:33 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:54:34 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/10 10:54:34 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:54:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/10 10:54:41 Waiting for app: TestDataTypes to get deployed
2021/12/10 10:54:46 Waiting for app: TestDataTypes to get deployed
2021/12/10 10:54:51 Waiting for app: TestDataTypes to get deployed
2021/12/10 10:54:56 Waiting for app: TestDataTypes to get deployed
2021/12/10 10:54:56 App: TestDataTypes got deployed
2021/12/10 10:54:56 Starting bucket ops to source bucket
2021/12/10 10:55:01 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/10 10:55:06 Waiting for dst bucket item count to get to: 18 curr count: 8
2021/12/10 10:55:06 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/10 10:55:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:55:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:55:11 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 10:55:16 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 10:55:21 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 10:55:26 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 10:55:26 App: TestDataTypes got un-deployed
2021/12/10 10:55:26 No eventing-con process running
2021/12/10 10:55:26 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/10 10:55:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:55:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:55:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:55:45 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/10 10:55:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (70.49s)
=== RUN   TestEnoentGet
2021/12/10 10:55:45 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:55:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/10 10:55:51 Waiting for app: TestEnoentGet to get deployed
2021/12/10 10:55:56 Waiting for app: TestEnoentGet to get deployed
2021/12/10 10:56:01 Waiting for app: TestEnoentGet to get deployed
2021/12/10 10:56:06 Waiting for app: TestEnoentGet to get deployed
2021/12/10 10:56:06 App: TestEnoentGet got deployed
2021/12/10 10:56:06 Starting bucket ops to source bucket
2021/12/10 10:56:11 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 10:56:11 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 10:56:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:56:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:56:16 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 10:56:21 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 10:56:26 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 10:56:31 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 10:56:31 App: TestEnoentGet got un-deployed
2021/12/10 10:56:31 No eventing-con process running
2021/12/10 10:56:32 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/10 10:56:33 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:56:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:56:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:56:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:56:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:57:00 Waiting for dst bucket item count to get to: 0 curr count: 64
2021/12/10 10:57:05 Waiting for dst bucket item count to get to: 0 curr count: 64
2021/12/10 10:57:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:57:05 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:57:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/10 10:57:11 Waiting for app: TestEnoentGet to get deployed
2021/12/10 10:57:16 Waiting for app: TestEnoentGet to get deployed
2021/12/10 10:57:21 Waiting for app: TestEnoentGet to get deployed
2021/12/10 10:57:26 Waiting for app: TestEnoentGet to get deployed
2021/12/10 10:57:26 App: TestEnoentGet got deployed
2021/12/10 10:57:26 Starting bucket ops to source bucket
2021/12/10 10:57:31 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 10:57:31 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 10:57:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:57:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:57:37 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 10:57:42 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 10:57:47 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 10:57:52 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 10:57:52 App: TestEnoentGet got un-deployed
2021/12/10 10:57:52 No eventing-con process running
2021/12/10 10:57:52 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/10 10:57:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:58:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:58:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:58:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:58:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:58:20 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/10 10:58:25 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/10 10:58:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.28s)
=== RUN   TestEnoentDelete
2021/12/10 10:58:25 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:58:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 10:58:31 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 10:58:36 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 10:58:41 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 10:58:46 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 10:58:46 App: TestEnoentDelete got deployed
2021/12/10 10:58:46 Starting bucket ops to source bucket
2021/12/10 10:58:52 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 10:58:52 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 10:58:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 10:58:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 10:58:57 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 10:59:02 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 10:59:07 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 10:59:12 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 10:59:12 App: TestEnoentDelete got un-deployed
2021/12/10 10:59:12 No eventing-con process running
2021/12/10 10:59:12 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/10 10:59:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:59:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 10:59:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 10:59:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:59:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 10:59:40 Waiting for dst bucket item count to get to: 0 curr count: 76
2021/12/10 10:59:45 Waiting for dst bucket item count to get to: 0 curr count: 76
2021/12/10 10:59:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 10:59:45 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 10:59:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 10:59:51 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 10:59:56 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 11:00:01 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 11:00:06 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 11:00:06 App: TestEnoentDelete got deployed
2021/12/10 11:00:06 Starting bucket ops to source bucket
2021/12/10 11:00:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 11:00:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 11:00:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:00:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:00:17 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 11:00:22 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 11:00:27 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 11:00:32 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 11:00:32 App: TestEnoentDelete got un-deployed
2021/12/10 11:00:32 No eventing-con process running
2021/12/10 11:00:32 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/10 11:00:33 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:00:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 11:00:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 11:00:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:00:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:01:00 Waiting for dst bucket item count to get to: 0 curr count: 76
2021/12/10 11:01:05 Waiting for dst bucket item count to get to: 0 curr count: 76
2021/12/10 11:01:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.08s)
=== RUN   TestError
2021/12/10 11:01:15 Waiting for app: TestError to get un-deployed
2021/12/10 11:01:15 App: TestError got un-deployed
2021/12/10 11:01:15 No eventing-con process running
2021/12/10 11:01:15 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/10 11:01:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:01:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:01:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:01:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:01:29 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:01:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/10 11:01:35 Waiting for app: TestError to get deployed
2021/12/10 11:01:40 Waiting for app: TestError to get deployed
2021/12/10 11:01:45 Waiting for app: TestError to get deployed
2021/12/10 11:01:50 Waiting for app: TestError to get deployed
2021/12/10 11:01:50 App: TestError got deployed
2021/12/10 11:01:50 Starting bucket ops to source bucket
2021/12/10 11:01:55 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/10 11:02:00 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/10 11:02:05 Waiting for dst bucket item count to get to: 300 curr count: 156
2021/12/10 11:02:05 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/10 11:02:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:02:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:02:10 Waiting for app: TestError to get un-deployed
2021/12/10 11:02:15 Waiting for app: TestError to get un-deployed
2021/12/10 11:02:20 Waiting for app: TestError to get un-deployed
2021/12/10 11:02:25 Waiting for app: TestError to get un-deployed
2021/12/10 11:02:25 App: TestError got un-deployed
2021/12/10 11:02:25 No eventing-con process running
2021/12/10 11:02:25 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/10 11:02:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:02:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 11:02:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:02:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:02:49 Waiting for dst bucket item count to get to: 0 curr count: 300
2021/12/10 11:02:54 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/10 11:02:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.66s)
=== RUN   TestCRLF
2021/12/10 11:03:04 Waiting for app: TestCRLF to get un-deployed
2021/12/10 11:03:04 App: TestCRLF got un-deployed
2021/12/10 11:03:04 No eventing-con process running
2021/12/10 11:03:04 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/10 11:03:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:03:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:03:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:03:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:03:18 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:03:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/10 11:03:24 Waiting for app: TestCRLF to get deployed
2021/12/10 11:03:29 Waiting for app: TestCRLF to get deployed
2021/12/10 11:03:34 Waiting for app: TestCRLF to get deployed
2021/12/10 11:03:39 Waiting for app: TestCRLF to get deployed
2021/12/10 11:03:39 App: TestCRLF got deployed
2021/12/10 11:03:39 Starting bucket ops to source bucket
2021/12/10 11:03:44 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 11:03:49 Waiting for dst bucket item count to get to: 100 curr count: 45
2021/12/10 11:03:49 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 11:03:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:03:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:03:54 Waiting for app: TestCRLF to get un-deployed
2021/12/10 11:03:59 Waiting for app: TestCRLF to get un-deployed
2021/12/10 11:04:04 Waiting for app: TestCRLF to get un-deployed
2021/12/10 11:04:09 Waiting for app: TestCRLF to get un-deployed
2021/12/10 11:04:09 App: TestCRLF got un-deployed
2021/12/10 11:04:09 No eventing-con process running
2021/12/10 11:04:09 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/10 11:04:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:04:21 Waiting for dst bucket item count to get to: 0 curr count: 99
2021/12/10 11:04:26 Waiting for dst bucket item count to get to: 0 curr count: 99
2021/12/10 11:04:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:04:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:04:38 Waiting for dst bucket item count to get to: 0 curr count: 51
2021/12/10 11:04:43 Waiting for dst bucket item count to get to: 0 curr count: 6
2021/12/10 11:04:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.78s)
=== RUN   TestImportExport
2021/12/10 11:04:53 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 11:04:53 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/10 11:04:53 No eventing-con process running
2021/12/10 11:04:53 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/10 11:04:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:04:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:05:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:05:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:05:11 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 11:05:11 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/10 11:05:11 No eventing-con process running
2021/12/10 11:05:11 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/10 11:05:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:05:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:05:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:05:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:05:25 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:05:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 11:05:26 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:05:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 11:05:32 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 11:05:37 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 11:05:42 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 11:05:47 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 11:05:47 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/10 11:05:52 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/10 11:05:57 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/10 11:06:02 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/10 11:06:02 App: TestImportExport_bucket_op_on_update got deployed
2021/12/10 11:06:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:06:08 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 11:06:13 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 11:06:18 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 11:06:23 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 11:06:23 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/10 11:06:23 No eventing-con process running
2021/12/10 11:06:23 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/10 11:06:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:06:28 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 11:06:33 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 11:06:38 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 11:06:43 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 11:06:43 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/10 11:06:43 No eventing-con process running
2021/12/10 11:06:43 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/10 11:07:00 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 11:07:00 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/10 11:07:00 No eventing-con process running
2021/12/10 11:07:00 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/10 11:07:02 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:07:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:07:09 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:07:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:07:19 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 11:07:19 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/10 11:07:19 No eventing-con process running
2021/12/10 11:07:19 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/10 11:07:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:07:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:07:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:07:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.64s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/10 11:07:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:07:42 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 11:07:42 No eventing-con process running
2021/12/10 11:07:42 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/10 11:07:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:07:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:07:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:07:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:07:56 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:07:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 11:07:57 Starting bucket ops to source bucket
2021/12/10 11:08:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:08:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:08:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:08:20 Waiting for dst bucket item count to get to: 5000 curr count: 995
2021/12/10 11:08:25 Waiting for dst bucket item count to get to: 5000 curr count: 995
2021/12/10 11:08:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:08:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 11:08:30 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 11:08:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:08:30 Undeploying app: bucket_op_on_update
2021/12/10 11:08:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:08:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:08:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:08:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:08:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:08:50 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 11:08:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:08:50 No eventing-con process running
2021/12/10 11:08:52 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:08:59 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:09:34 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:09:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 11:09:35 Starting bucket ops to source bucket
2021/12/10 11:09:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:09:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:09:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:09:58 Waiting for dst bucket item count to get to: 5000 curr count: 3098
2021/12/10 11:10:03 Waiting for dst bucket item count to get to: 5000 curr count: 3098
2021/12/10 11:10:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:10:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 11:10:08 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 11:10:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:10:08 Undeploying app: bucket_op_on_update
2021/12/10 11:10:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:10:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:10:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:10:23 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:10:28 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:10:28 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 11:10:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:10:28 No eventing-con process running
2021/12/10 11:10:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:10:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:11:12 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:11:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 11:11:14 Starting bucket ops to source bucket
2021/12/10 11:11:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:11:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:11:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:11:37 Waiting for dst bucket item count to get to: 5000 curr count: 2428
2021/12/10 11:11:42 Waiting for dst bucket item count to get to: 5000 curr count: 4928
2021/12/10 11:11:42 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:11:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 11:11:47 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 11:11:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:11:47 Undeploying app: bucket_op_on_update
2021/12/10 11:11:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:11:52 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:11:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:12:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:12:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:12:07 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 11:12:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:12:07 No eventing-con process running
2021/12/10 11:12:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:12:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:12:51 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:12:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 11:12:52 Starting bucket ops to source bucket
2021/12/10 11:13:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:13:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:13:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:13:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:13:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:13:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 11:13:20 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 11:13:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:13:20 Undeploying app: bucket_op_on_update
2021/12/10 11:13:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:13:25 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:13:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:13:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:13:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:13:40 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 11:13:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:13:40 No eventing-con process running
2021/12/10 11:13:42 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:13:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:14:24 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:14:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 11:14:25 Starting bucket ops to source bucket
2021/12/10 11:14:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:14:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:14:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:14:48 Waiting for dst bucket item count to get to: 5000 curr count: 2664
2021/12/10 11:14:53 Waiting for dst bucket item count to get to: 5000 curr count: 2664
2021/12/10 11:14:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:14:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 11:14:58 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 11:14:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:14:58 Undeploying app: bucket_op_on_update
2021/12/10 11:14:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:15:03 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:15:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:15:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:15:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 11:15:18 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 11:15:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:15:18 No eventing-con process running
2021/12/10 11:15:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:15:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:16:02 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (509.97s)
=== RUN   TestOnUpdateN1QLOp
2021/12/10 11:16:12 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 11:16:12 App: TestOnUpdateN1QLOp got un-deployed
2021/12/10 11:16:12 No eventing-con process running
2021/12/10 11:16:12 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/10 11:16:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:16:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:16:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:16:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:16:26 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:16:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/10 11:16:27 Starting bucket ops to source bucket
2021/12/10 11:16:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:16:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:16:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:16:50 Waiting for dst bucket item count to get to: 5000 curr count: 253
2021/12/10 11:16:55 Waiting for dst bucket item count to get to: 5000 curr count: 253
2021/12/10 11:16:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:16:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:16:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:17:00 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 11:17:05 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 11:17:10 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 11:17:15 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 11:17:15 App: TestOnUpdateN1QLOp got un-deployed
2021/12/10 11:17:15 No eventing-con process running
2021/12/10 11:17:15 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/10 11:17:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:17:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 11:17:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:17:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:17:39 Waiting for dst bucket item count to get to: 0 curr count: 4597
2021/12/10 11:17:44 Waiting for dst bucket item count to get to: 0 curr count: 2136
2021/12/10 11:17:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.28s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/10 11:17:54 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 11:17:54 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/10 11:17:54 No eventing-con process running
2021/12/10 11:17:54 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/10 11:17:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:18:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:18:02 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:18:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:18:07 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:18:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 11:18:08 Starting bucket ops to source bucket
2021/12/10 11:18:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:18:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:18:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:18:31 Waiting for dst bucket item count to get to: 5000 curr count: 55
2021/12/10 11:18:36 Waiting for dst bucket item count to get to: 5000 curr count: 2555
2021/12/10 11:18:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:18:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:18:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:18:41 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 11:18:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 11:18:51 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 11:18:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 11:18:56 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/10 11:18:56 No eventing-con process running
2021/12/10 11:18:56 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/10 11:18:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:19:08 Waiting for dst bucket item count to get to: 0 curr count: 3742
2021/12/10 11:19:13 Waiting for dst bucket item count to get to: 0 curr count: 1242
2021/12/10 11:19:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:19:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:19:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 11:19:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.43s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/10 11:19:35 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 11:19:35 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 11:19:35 No eventing-con process running
2021/12/10 11:19:35 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/10 11:19:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:19:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:19:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:19:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:19:49 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:19:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 11:19:50 Starting bucket ops to source bucket
2021/12/10 11:19:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:20:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:20:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:20:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:20:18 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 11:20:18 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:20:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:20:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:20:23 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 11:20:28 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 11:20:33 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 11:20:38 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 11:20:38 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 11:20:38 No eventing-con process running
2021/12/10 11:20:38 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/10 11:20:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:20:50 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 11:20:55 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 11:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:20:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:21:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 11:21:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.55s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/10 11:21:17 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 11:21:17 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/10 11:21:17 No eventing-con process running
2021/12/10 11:21:17 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/10 11:21:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:21:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:21:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:21:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:21:30 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:21:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/10 11:21:31 Starting bucket ops to source bucket
2021/12/10 11:21:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:21:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:21:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:21:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:21:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:21:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:21:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:21:59 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 11:22:04 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 11:22:09 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 11:22:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 11:22:14 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/10 11:22:14 No eventing-con process running
2021/12/10 11:22:14 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/10 11:22:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:22:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 11:22:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:22:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:22:38 Waiting for dst bucket item count to get to: 0 curr count: 3632
2021/12/10 11:22:43 Waiting for dst bucket item count to get to: 0 curr count: 1132
2021/12/10 11:22:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (96.47s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/10 11:22:53 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 11:22:53 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/10 11:22:53 No eventing-con process running
2021/12/10 11:22:53 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/10 11:22:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:23:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:23:02 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:23:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:23:07 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:23:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/10 11:23:08 Starting bucket ops to source bucket
2021/12/10 11:23:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:23:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:23:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:23:31 Waiting for dst bucket item count to get to: 5000 curr count: 207
2021/12/10 11:23:36 Waiting for dst bucket item count to get to: 5000 curr count: 207
2021/12/10 11:23:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:23:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:23:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:23:41 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 11:23:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 11:23:51 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 11:23:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 11:23:56 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/10 11:23:57 No eventing-con process running
2021/12/10 11:23:57 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/10 11:23:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:24:08 Waiting for dst bucket item count to get to: 0 curr count: 3788
2021/12/10 11:24:13 Waiting for dst bucket item count to get to: 0 curr count: 1288
2021/12/10 11:24:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:24:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:24:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 11:24:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.08s)
=== RUN   TestOnDeleteBucketOp
2021/12/10 11:24:35 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 11:24:35 App: TestOnDeleteBucketOp got un-deployed
2021/12/10 11:24:35 No eventing-con process running
2021/12/10 11:24:35 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/10 11:24:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:24:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:24:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:24:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:24:49 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:24:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 11:24:50 Starting bucket ops to source bucket
2021/12/10 11:25:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:25:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:25:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:25:16 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/10 11:25:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:25:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:25:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:25:21 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 11:25:26 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 11:25:31 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 11:25:36 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 11:25:36 App: TestOnDeleteBucketOp got un-deployed
2021/12/10 11:25:36 No eventing-con process running
2021/12/10 11:25:36 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/10 11:25:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:25:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:25:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:25:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 11:25:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (89.85s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/10 11:26:05 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 11:26:05 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/10 11:26:05 No eventing-con process running
2021/12/10 11:26:05 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/10 11:26:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:26:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:26:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:26:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:26:19 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:26:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/10 11:26:20 Starting bucket ops to source bucket
2021/12/10 11:26:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:26:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:26:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:26:45 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/10 11:26:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:26:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:26:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:26:51 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 11:26:56 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 11:27:01 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 11:27:06 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 11:27:06 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/10 11:27:06 No eventing-con process running
2021/12/10 11:27:06 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/10 11:27:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:27:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:27:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:27:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 11:27:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.73s)
=== RUN   TestTimerBucketOp
2021/12/10 11:27:35 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 11:27:35 App: TestTimerBucketOp got un-deployed
2021/12/10 11:27:35 No eventing-con process running
2021/12/10 11:27:35 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/10 11:27:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:27:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:27:43 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:27:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:27:48 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:27:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 11:27:49 Starting bucket ops to source bucket
2021/12/10 11:27:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:28:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:28:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:28:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:28:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:28:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:28:27 Waiting for dst bucket item count to get to: 5000 curr count: 1392
2021/12/10 11:28:32 Waiting for dst bucket item count to get to: 5000 curr count: 3892
2021/12/10 11:28:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:28:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:28:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:28:37 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 11:28:43 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 11:28:48 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 11:28:53 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 11:28:53 App: TestTimerBucketOp got un-deployed
2021/12/10 11:28:53 No eventing-con process running
2021/12/10 11:28:53 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/10 11:28:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:29:04 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 11:29:09 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 11:29:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:29:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:29:21 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 11:29:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 11:29:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (121.38s)
=== RUN   TestTimerInPastBucketOp
2021/12/10 11:29:36 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 11:29:36 App: TestTimerInPastBucketOp got un-deployed
2021/12/10 11:29:36 No eventing-con process running
2021/12/10 11:29:36 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/10 11:29:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:29:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:29:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:29:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:29:50 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:29:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 11:29:51 Starting bucket ops to source bucket
2021/12/10 11:29:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:30:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:30:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:30:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:30:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:30:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:30:29 Waiting for dst bucket item count to get to: 5000 curr count: 2825
2021/12/10 11:30:34 Waiting for dst bucket item count to get to: 5000 curr count: 2825
2021/12/10 11:30:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:30:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:30:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:30:39 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 11:30:44 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 11:30:49 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 11:30:54 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 11:30:54 App: TestTimerInPastBucketOp got un-deployed
2021/12/10 11:30:54 No eventing-con process running
2021/12/10 11:30:54 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/10 11:30:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:31:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 11:31:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:31:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:31:18 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 11:31:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.49s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/10 11:31:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:31:28 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 11:31:28 No eventing-con process running
2021/12/10 11:31:28 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/10 11:31:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:31:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:31:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:31:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:31:41 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:31:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 11:31:42 Starting bucket ops to source bucket
2021/12/10 11:31:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:31:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:32:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:32:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:32:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:32:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:32:20 Waiting for dst bucket item count to get to: 5000 curr count: 2786
2021/12/10 11:32:25 Waiting for dst bucket item count to get to: 5000 curr count: 2786
2021/12/10 11:32:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:32:30 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 11:32:30 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 11:32:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:32:30 Undeploying app: bucket_op_with_timer
2021/12/10 11:32:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:32:35 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:32:40 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:32:45 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:32:50 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:32:50 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 11:32:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:32:50 No eventing-con process running
2021/12/10 11:32:52 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:32:59 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:33:34 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:33:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 11:33:35 Starting bucket ops to source bucket
2021/12/10 11:33:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:33:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:33:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:33:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:34:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:34:08 Waiting for dst bucket item count to get to: 5000 curr count: 1018
2021/12/10 11:34:13 Waiting for dst bucket item count to get to: 5000 curr count: 1018
2021/12/10 11:34:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:34:18 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 11:34:18 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 11:34:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:34:19 Undeploying app: bucket_op_with_timer
2021/12/10 11:34:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:34:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:34:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:34:34 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:34:39 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:34:39 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 11:34:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:34:39 No eventing-con process running
2021/12/10 11:34:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:34:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:35:23 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:35:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 11:35:24 Starting bucket ops to source bucket
2021/12/10 11:35:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:35:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:35:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:35:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:35:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:35:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:36:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:36:07 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 11:36:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:36:12 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 11:36:12 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 11:36:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:36:12 Undeploying app: bucket_op_with_timer
2021/12/10 11:36:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:36:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:36:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:36:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:36:32 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:36:32 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 11:36:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:36:32 No eventing-con process running
2021/12/10 11:36:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:36:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:37:16 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:37:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 11:37:17 Starting bucket ops to source bucket
2021/12/10 11:37:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:37:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:37:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:37:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:37:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:37:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:37:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:37:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:38:00 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 11:38:00 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 11:38:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:38:00 Undeploying app: bucket_op_with_timer
2021/12/10 11:38:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:38:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:38:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:38:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:38:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:38:20 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 11:38:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:38:20 No eventing-con process running
2021/12/10 11:38:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:38:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:39:04 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:39:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 11:39:05 Starting bucket ops to source bucket
2021/12/10 11:39:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:39:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:39:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:39:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:39:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:39:38 Waiting for dst bucket item count to get to: 5000 curr count: 22
2021/12/10 11:39:43 Waiting for dst bucket item count to get to: 5000 curr count: 22
2021/12/10 11:39:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:39:48 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 11:39:48 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 11:39:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:39:48 Undeploying app: bucket_op_with_timer
2021/12/10 11:39:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:39:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:39:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:40:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:40:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 11:40:08 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 11:40:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:40:08 No eventing-con process running
2021/12/10 11:40:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:40:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:40:53 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (575.00s)
=== RUN   TestMultipleHandlers
2021/12/10 11:41:03 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 11:41:03 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/10 11:41:03 No eventing-con process running
2021/12/10 11:41:03 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/10 11:41:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:41:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:41:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:41:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:41:21 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 11:41:21 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/10 11:41:21 No eventing-con process running
2021/12/10 11:41:21 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/10 11:41:23 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:41:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:41:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:41:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:41:35 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:41:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 11:41:36 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:41:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/10 11:41:42 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/10 11:41:47 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/10 11:41:52 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/10 11:41:57 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/10 11:41:57 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/10 11:42:02 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/10 11:42:07 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/10 11:42:12 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/10 11:42:12 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/10 11:42:12 Starting bucket ops to source bucket
2021/12/10 11:42:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:42:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:42:22 Starting bucket ops to source bucket
2021/12/10 11:42:37 Waiting for dst bucket item count to get to: 10000 curr count: 6272
2021/12/10 11:42:42 Waiting for dst bucket item count to get to: 10000 curr count: 6272
2021/12/10 11:42:42 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 11:42:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:42:42 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:42:42 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:42:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:42:47 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 11:42:52 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 11:42:57 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 11:43:02 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 11:43:02 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/10 11:43:02 No eventing-con process running
2021/12/10 11:43:02 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/10 11:43:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:43:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:43:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:43:21 Waiting for dst bucket item count to get to: 0 curr count: 9498
2021/12/10 11:43:26 Waiting for dst bucket item count to get to: 0 curr count: 9498
2021/12/10 11:43:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:43:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:43:31 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 11:43:36 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 11:43:41 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 11:43:47 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 11:43:47 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/10 11:43:47 No eventing-con process running
2021/12/10 11:43:47 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/10 11:43:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:43:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:43:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:44:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (187.58s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/10 11:44:10 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 11:44:10 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/10 11:44:10 No eventing-con process running
2021/12/10 11:44:10 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/10 11:44:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:44:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:44:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:44:24 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:44:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 11:44:25 Starting bucket ops to source bucket
2021/12/10 11:44:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:44:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:44:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:44:48 Waiting for dst bucket item count to get to: 5000 curr count: 1416
2021/12/10 11:44:53 Waiting for dst bucket item count to get to: 5000 curr count: 3916
2021/12/10 11:44:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:44:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:44:58 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 11:44:58 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 11:44:58 Pausing app: bucket_op_on_update
2021/12/10 11:44:58 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:45:03 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 11:45:03 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 11:45:03 Resuming app: bucket_op_on_update
2021/12/10 11:45:04 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:45:04 Starting bucket ops to source bucket
2021/12/10 11:45:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:45:17 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:45:22 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:45:22 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 11:45:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:45:27 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 11:45:27 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 11:45:27 Pausing app: bucket_op_on_update
2021/12/10 11:45:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:45:32 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 11:45:32 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 11:45:32 Resuming app: bucket_op_on_update
2021/12/10 11:45:33 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:45:33 Starting bucket ops to source bucket
2021/12/10 11:45:41 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 11:45:46 Waiting for dst bucket item count to get to: 15000 curr count: 10138
2021/12/10 11:45:51 Waiting for dst bucket item count to get to: 15000 curr count: 10138
2021/12/10 11:45:51 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/10 11:45:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:45:56 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 11:45:56 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 11:45:56 Pausing app: bucket_op_on_update
2021/12/10 11:45:56 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:46:01 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 11:46:01 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 11:46:01 Resuming app: bucket_op_on_update
2021/12/10 11:46:02 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:46:02 Starting bucket ops to source bucket
2021/12/10 11:46:10 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 11:46:15 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 11:46:20 Waiting for dst bucket item count to get to: 20000 curr count: 15577
2021/12/10 11:46:25 Waiting for dst bucket item count to get to: 20000 curr count: 15577
2021/12/10 11:46:25 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/10 11:46:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:46:30 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 11:46:30 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 11:46:30 Pausing app: bucket_op_on_update
2021/12/10 11:46:31 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:46:36 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 11:46:36 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 11:46:36 Resuming app: bucket_op_on_update
2021/12/10 11:46:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:46:37 Starting bucket ops to source bucket
2021/12/10 11:46:45 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 11:46:50 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 11:46:55 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 11:46:55 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/10 11:46:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:47:00 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 11:47:00 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 11:47:00 Pausing app: bucket_op_on_update
2021/12/10 11:47:00 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:47:05 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 11:47:05 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 11:47:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:47:10 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 11:47:15 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 11:47:20 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 11:47:25 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 11:47:25 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/10 11:47:25 No eventing-con process running
2021/12/10 11:47:25 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/10 11:47:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:47:37 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 11:47:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:47:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:47:49 Waiting for dst bucket item count to get to: 0 curr count: 23032
2021/12/10 11:47:54 Waiting for dst bucket item count to get to: 0 curr count: 10532
2021/12/10 11:47:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.55s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/10 11:48:04 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 11:48:04 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/10 11:48:04 No eventing-con process running
2021/12/10 11:48:04 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/10 11:48:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:48:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:48:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:48:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:48:17 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:48:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 11:48:18 Starting bucket ops to source bucket
2021/12/10 11:48:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:48:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:48:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:48:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:48:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:48:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:48:56 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 11:48:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:48:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:49:01 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 11:49:01 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 11:49:01 Pausing app: bucket_op_with_timer
2021/12/10 11:49:02 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:49:07 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 11:49:07 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 11:49:07 Resuming app: bucket_op_with_timer
2021/12/10 11:49:08 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:49:08 Starting bucket ops to source bucket
2021/12/10 11:49:16 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:49:21 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:49:26 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:49:31 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:49:36 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:49:41 Waiting for dst bucket item count to get to: 10000 curr count: 5437
2021/12/10 11:49:46 Waiting for dst bucket item count to get to: 10000 curr count: 7937
2021/12/10 11:49:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 11:49:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:49:51 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 11:49:51 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 11:49:51 Pausing app: bucket_op_with_timer
2021/12/10 11:49:51 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:49:56 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 11:49:56 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 11:49:56 Resuming app: bucket_op_with_timer
2021/12/10 11:49:57 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:49:57 Starting bucket ops to source bucket
2021/12/10 11:50:05 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 11:50:10 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 11:50:15 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 11:50:20 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 11:50:25 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 11:50:30 Waiting for dst bucket item count to get to: 15000 curr count: 11558
2021/12/10 11:50:35 Waiting for dst bucket item count to get to: 15000 curr count: 14058
2021/12/10 11:50:35 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/10 11:50:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:50:40 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 11:50:40 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 11:50:40 Pausing app: bucket_op_with_timer
2021/12/10 11:50:40 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:50:45 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 11:50:45 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 11:50:45 Resuming app: bucket_op_with_timer
2021/12/10 11:50:46 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:50:46 Starting bucket ops to source bucket
2021/12/10 11:50:54 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 11:50:59 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 11:51:04 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 11:51:09 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 11:51:14 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 11:51:19 Waiting for dst bucket item count to get to: 20000 curr count: 17420
2021/12/10 11:51:24 Waiting for dst bucket item count to get to: 20000 curr count: 17420
2021/12/10 11:51:24 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/10 11:51:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:51:29 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 11:51:29 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 11:51:29 Pausing app: bucket_op_with_timer
2021/12/10 11:51:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:51:35 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 11:51:35 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 11:51:35 Resuming app: bucket_op_with_timer
2021/12/10 11:51:36 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:51:36 Starting bucket ops to source bucket
2021/12/10 11:51:44 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 11:51:49 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 11:51:54 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 11:51:59 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 11:52:04 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 11:52:09 Waiting for dst bucket item count to get to: 25000 curr count: 22927
2021/12/10 11:52:14 Waiting for dst bucket item count to get to: 25000 curr count: 22927
2021/12/10 11:52:14 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/10 11:52:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:52:19 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 11:52:19 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 11:52:19 Pausing app: bucket_op_with_timer
2021/12/10 11:52:19 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:52:24 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 11:52:24 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 11:52:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:52:29 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 11:52:34 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 11:52:39 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 11:52:44 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 11:52:44 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/10 11:52:44 No eventing-con process running
2021/12/10 11:52:44 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/10 11:52:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:52:56 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 11:52:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:52:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:53:08 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/10 11:53:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (313.82s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/10 11:53:18 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 11:53:18 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/10 11:53:18 No eventing-con process running
2021/12/10 11:53:18 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/10 11:53:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:53:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:53:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:53:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:53:31 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:53:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/10 11:53:37 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 11:53:42 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 11:53:47 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 11:53:52 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 11:53:52 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/10 11:53:52 Starting bucket ops to source bucket
2021/12/10 11:54:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:54:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:54:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:54:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:54:22 Waiting for dst bucket item count to get to: 5000 curr count: 4148
2021/12/10 11:54:27 Waiting for dst bucket item count to get to: 5000 curr count: 4148
2021/12/10 11:54:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:54:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:54:27 Pausing app: bucket_op_with_timer
2021/12/10 11:54:27 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:54:32 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/10 11:54:32 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/10 11:54:32 Resuming app: bucket_op_with_timer
2021/12/10 11:54:33 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:54:33 Starting bucket ops to source bucket
2021/12/10 11:54:45 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:54:50 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:54:55 Waiting for dst bucket item count to get to: 10000 curr count: 5013
2021/12/10 11:55:00 Waiting for dst bucket item count to get to: 10000 curr count: 5106
2021/12/10 11:55:05 Waiting for dst bucket item count to get to: 10000 curr count: 7079
2021/12/10 11:55:10 Waiting for dst bucket item count to get to: 10000 curr count: 9486
2021/12/10 11:55:10 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 11:55:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 11:55:15 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 11:55:20 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 11:55:25 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 11:55:30 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 11:55:30 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/10 11:55:30 No eventing-con process running
2021/12/10 11:55:30 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/10 11:55:32 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:55:42 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 11:55:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 11:55:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:55:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:55:58 Waiting for dst bucket item count to get to: 0 curr count: 8402
2021/12/10 11:56:03 Waiting for dst bucket item count to get to: 0 curr count: 3402
2021/12/10 11:56:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (175.86s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/10 11:56:13 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 11:56:13 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/10 11:56:13 No eventing-con process running
2021/12/10 11:56:13 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/10 11:56:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:56:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:56:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 11:56:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 11:56:27 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 11:56:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/10 11:56:33 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 11:56:38 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 11:56:43 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 11:56:48 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 11:56:48 App: TestPauseResumeWithEventingReb got deployed
2021/12/10 11:56:48 Starting bucket ops to source bucket
2021/12/10 11:56:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:57:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:57:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:57:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 11:57:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 11:57:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 11:57:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 11:57:17 Pausing app: bucket_op_with_timer
2021/12/10 11:57:18 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:57:23 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/10 11:57:23 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/10 11:57:23 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 11:57:23 Removing node(s): [] from the cluster
2021/12/10 11:57:28 Rebalance progress: 29.16666666666666
2021/12/10 11:57:33 Rebalance progress: 29.16666666666666
2021/12/10 11:57:38 Rebalance progress: 100
2021/12/10 11:57:38 Resuming app: bucket_op_with_timer
2021/12/10 11:57:39 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 11:57:39 Starting bucket ops to source bucket
2021/12/10 11:57:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:57:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:58:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:58:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:58:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 11:58:16 Waiting for dst bucket item count to get to: 10000 curr count: 7365
2021/12/10 11:58:21 Waiting for dst bucket item count to get to: 10000 curr count: 9865
2021/12/10 11:58:21 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 11:58:21 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 11:58:26 Rebalance progress: 29.16666666666666
2021/12/10 11:58:31 Rebalance progress: 29.16666666666666
2021/12/10 11:58:36 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/10 11:58:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 11:58:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 11:59:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 11:59:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 11:59:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 11:59:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 12:00:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 12:00:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 12:00:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 12:00:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 12:01:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 12:01:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 12:01:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 12:01:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: bdaa326d7c24af83337ce72a470d792f
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
Producer node: 172.31.2.134:9300	Node UUID: ae5c12a6d0163652e7b530be4e677ac7
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
Producer node: 127.0.0.1:9303	Node UUID: 8412b11f9f4b861c67c28e98cbac77fc
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]


2021/12/10 12:02:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:02:11 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:02:16 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 12:02:21 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 12:02:26 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 12:02:31 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 12:02:31 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/10 12:02:31 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:02:31 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/10 12:02:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:02:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 12:02:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:02:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:02:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:03:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:03:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:03:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (421.23s)
=== RUN   TestBucketDeleteAfterPause
2021/12/10 12:03:15 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/10 12:03:15 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/10 12:03:15 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:03:15 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/10 12:03:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:03:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:03:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:03:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:03:28 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:03:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/10 12:03:34 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 12:03:39 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 12:03:44 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 12:03:49 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 12:03:49 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/10 12:03:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:03:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:04:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:04:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 12:04:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 12:04:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:04:09 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/10 12:04:09 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 12:04:14 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/10 12:04:14 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/10 12:04:14 handler_test.go:716: Deleting source bucket: default
2021/12/10 12:04:20 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/10 12:04:21 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/10 12:04:26 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/10 12:04:26 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/10 12:04:26 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:04:26 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/10 12:04:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:04:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 12:04:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:04:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:04:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:04:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 12:04:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (109.62s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/10 12:05:04 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 12:05:04 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/10 12:05:04 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:05:04 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/10 12:05:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:05:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:05:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:05:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:05:17 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:05:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/10 12:05:23 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 12:05:28 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 12:05:33 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 12:05:38 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 12:05:38 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/10 12:05:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:06:13 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/10 12:06:13 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 12:06:18 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/10 12:06:18 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/10 12:09:18 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/10 12:09:18 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/10 12:09:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/10 12:09:24 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 12:09:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:09:24 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/10 12:09:29 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 12:09:30 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:09:30 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3312 expected 5000
2021/12/10 12:09:35 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 12:09:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:09:35 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 4452 expected 5000
2021/12/10 12:09:40 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 12:09:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:09:40 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/10 12:09:40 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/10 12:09:40 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 12:09:50 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 12:09:55 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 12:10:00 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 12:10:00 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/10 12:10:00 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:10:00 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/10 12:10:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:10:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:10:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:10:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:10:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:10:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.54s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/10 12:10:33 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 12:10:33 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/10 12:10:33 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:10:33 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/10 12:10:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:10:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:10:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:10:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:10:46 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:10:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 12:10:52 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 12:10:57 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 12:11:02 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 12:11:07 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 12:11:07 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/10 12:11:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:11:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:11:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1535
2021/12/10 12:11:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1535
2021/12/10 12:11:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 12:11:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:11:26 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/10 12:11:26 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 12:11:56 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/10 12:11:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 12:12:02 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 12:12:07 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 12:12:12 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 12:12:17 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 12:12:17 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/10 12:12:17 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:12:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:12:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 12:12:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:12:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:12:35 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/10 12:12:35 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 12:12:45 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 12:12:50 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 12:12:55 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 12:12:55 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/10 12:12:55 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:12:55 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/10 12:12:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:13:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:13:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:13:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.21s)
=== RUN   TestCPPWorkerCleanup
2021/12/10 12:13:18 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 12:13:18 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/10 12:13:18 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:13:18 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/10 12:13:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:13:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:13:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:13:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:13:31 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:13:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/10 12:13:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:13:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:13:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:13:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:13:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:13:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2932
2021/12/10 12:14:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2932
2021/12/10 12:14:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 12:14:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:14:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:14:09 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 12:14:14 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 12:14:20 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 12:14:25 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 12:14:25 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/10 12:14:25 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:14:25 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/10 12:14:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:14:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:14:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:14:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:14:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 12:14:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (129.57s)
=== RUN   TestWithUserXattrs
2021/12/10 12:15:28 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 12:15:28 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/10 12:15:28 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:15:28 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/10 12:15:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:15:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:15:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:15:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:15:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:15:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:15:44 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:15:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/10 12:15:51 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 12:15:56 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 12:16:01 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 12:16:06 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 12:16:06 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/10 12:16:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:16:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:16:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3171
2021/12/10 12:16:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3171
2021/12/10 12:16:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 12:16:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:16:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:16:30 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 12:16:35 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 12:16:40 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 12:16:45 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 12:16:45 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/10 12:16:45 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:16:45 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/10 12:16:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:16:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:16:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:16:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:17:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3320
2021/12/10 12:17:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 820
2021/12/10 12:17:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (115.38s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/10 12:17:23 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 12:17:23 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/10 12:17:23 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:17:23 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/10 12:17:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:17:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:17:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:17:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:17:36 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:17:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/10 12:17:42 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 12:17:47 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 12:17:52 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 12:17:57 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 12:17:57 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/10 12:17:57 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:18:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:18:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1283
2021/12/10 12:18:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1283
2021/12/10 12:18:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 12:18:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:18:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:18:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1746
2021/12/10 12:18:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1746
2021/12/10 12:18:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:18:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:18:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:18:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:18:47 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:19:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 13944
2021/12/10 12:19:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 20144
2021/12/10 12:19:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/10 12:19:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:19:17 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 12:19:22 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 12:19:27 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 12:19:32 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 12:19:37 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 12:19:37 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/10 12:19:37 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:19:42 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 12:19:42 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/10 12:19:42 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:19:42 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/10 12:19:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:19:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 12:19:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/10 12:19:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:20:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:20:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 16958
2021/12/10 12:20:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 16958
2021/12/10 12:20:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (182.16s)
=== RUN   TestMetaBucketDelete
2021/12/10 12:20:25 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/10 12:20:25 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/10 12:20:25 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:20:25 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/10 12:20:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:20:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:20:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:20:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:20:38 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:20:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 12:20:44 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 12:20:49 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 12:20:54 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 12:20:59 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 12:20:59 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/10 12:20:59 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/10 12:21:01 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/10 12:21:06 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/10 12:21:06 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/10 12:21:21 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/10 12:21:21 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/10 12:21:21 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:21:21 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/10 12:21:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:21:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:21:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:21:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.87s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/10 12:21:44 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 12:21:44 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 12:21:44 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:21:44 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/10 12:21:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:21:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:21:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:21:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:21:57 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:21:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 12:22:17 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/10 12:22:18 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/10 12:22:33 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 12:22:33 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 12:22:48 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 12:22:48 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 12:22:48 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:22:48 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/10 12:22:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:22:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:22:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:23:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (87.00s)
=== RUN   TestSourceBucketDelete
2021/12/10 12:23:11 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/10 12:23:11 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/10 12:23:11 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:23:11 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/10 12:23:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:23:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:23:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:23:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:23:24 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:23:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 12:23:30 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 12:23:35 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 12:23:40 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 12:23:45 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 12:23:45 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/10 12:23:45 handler_test.go:945: Deleting source bucket: default
2021/12/10 12:23:47 handler_test.go:947: Deleted source bucket: default
2021/12/10 12:23:52 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/10 12:23:52 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/10 12:24:07 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/10 12:24:07 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/10 12:24:07 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:24:07 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/10 12:24:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:24:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:24:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:24:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.09s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/10 12:24:30 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/10 12:24:30 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/10 12:24:30 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:24:30 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/10 12:24:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:24:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:24:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:24:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:24:43 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:24:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 12:25:03 handler_test.go:966: Deleting source bucket: default
2021/12/10 12:25:05 handler_test.go:968: Deleted source bucket: default
2021/12/10 12:25:20 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/10 12:25:20 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/10 12:25:35 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/10 12:25:35 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/10 12:25:35 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:25:35 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/10 12:25:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:25:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:25:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:25:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.77s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/10 12:25:58 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 12:25:58 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 12:25:58 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:25:58 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/10 12:25:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:26:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:26:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:26:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:26:11 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:26:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 12:26:31 handler_test.go:988: Deleting source bucket: default
2021/12/10 12:26:32 handler_test.go:990: Deleted source bucket: default
2021/12/10 12:26:32 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/10 12:26:33 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/10 12:26:48 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 12:26:48 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 12:27:03 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 12:27:03 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 12:27:03 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:27:03 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/10 12:27:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:27:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:27:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:27:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.12s)
=== RUN   TestUndeployDuringBootstrap
2021/12/10 12:27:26 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 12:27:26 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/10 12:27:26 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:27:26 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/10 12:27:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:27:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:27:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:27:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:27:39 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:27:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 12:27:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:27:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:27:52 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:27:52 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/10 12:27:52 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/10 12:27:52 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/10 12:27:57 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/10 12:28:02 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/10 12:28:07 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 12:28:12 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 12:28:17 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 12:28:17 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/10 12:28:17 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:28:42 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 12:28:42 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/10 12:28:43 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:28:43 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/10 12:28:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:28:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:28:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 12:28:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:29:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:29:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2936
2021/12/10 12:29:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2936
2021/12/10 12:29:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.57s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/10 12:29:21 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:29:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/10 12:29:27 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 12:29:32 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 12:29:37 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 12:29:42 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 12:29:42 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/10 12:29:42 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/10 12:29:42 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 12:29:42 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/10 12:29:47 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/10 12:29:47 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/10 12:29:47 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:29:47 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/10 12:29:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:29:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:29:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:30:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.56s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/10 12:30:10 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 12:30:10 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/10 12:30:10 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:30:10 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/10 12:30:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:30:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:30:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:30:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:30:23 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:30:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/10 12:30:30 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 12:30:35 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 12:30:40 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 12:30:45 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 12:30:45 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/10 12:30:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:31:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:31:20 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 12:31:25 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 12:31:30 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 12:31:35 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 12:31:35 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/10 12:31:35 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:33:20 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 12:33:20 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/10 12:33:20 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:33:20 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/10 12:33:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:33:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:33:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:33:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:33:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:33:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3696
2021/12/10 12:33:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1196
2021/12/10 12:33:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (233.03s)
=== RUN   TestUndeployWithKVFailover
2021/12/10 12:34:03 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/10 12:34:03 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/10 12:34:03 eventingSetup.go:46: No eventing-con process running
2021/12/10 12:34:03 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/10 12:34:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:34:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:34:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:34:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:34:16 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:34:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/10 12:34:17 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 12:34:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:34:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:34:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:34:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:34:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:34:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:34:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:34:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 12:34:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 12:34:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 12:35:00 Rebalance progress: 29.16666666666666
2021/12/10 12:35:05 Rebalance progress: 29.16666666666666
2021/12/10 12:35:10 Rebalance progress: 100
2021/12/10 12:35:10 Failing over: [127.0.0.1:19003]
2021/12/10 12:35:20 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 12:36:20 Removing node(s): [] from the cluster
2021/12/10 12:36:25 Rebalance progress: 29.16666666666666
2021/12/10 12:36:30 Rebalance progress: 29.16666666666666
2021/12/10 12:36:35 Rebalance progress: 100
2021/12/10 12:36:40 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/10 12:36:40 App: TestUndeployWithKVFailover got un-deployed
2021/12/10 12:36:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:36:45 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/10 12:36:45 App: TestUndeployWithKVFailover got un-deployed
2021/12/10 12:36:45 No eventing-con process running
2021/12/10 12:36:45 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/10 12:36:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:36:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:36:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:36:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:37:08 Waiting for dst bucket item count to get to: 0 curr count: 3278
2021/12/10 12:37:13 Waiting for dst bucket item count to get to: 0 curr count: 778
2021/12/10 12:37:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (199.83s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/10 12:37:23 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 12:37:23 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/10 12:37:23 No eventing-con process running
2021/12/10 12:37:23 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/10 12:37:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:37:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:37:31 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:37:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:37:36 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:37:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/10 12:37:42 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 12:37:47 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 12:37:52 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 12:37:57 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 12:37:57 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/10 12:37:57 Starting bucket ops to source bucket
2021/12/10 12:38:18 Waiting for dst bucket item count to get to: 20000 curr count: 14334
2021/12/10 12:38:23 Waiting for dst bucket item count to get to: 20000 curr count: 14334
2021/12/10 12:38:23 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/10 12:38:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:38:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:38:36 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/10 12:38:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:38:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:38:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 12:38:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:38:47 Starting bucket ops to source bucket
2021/12/10 12:39:00 Waiting for dst bucket item count to get to: 10000 curr count: 1126
2021/12/10 12:39:05 Waiting for dst bucket item count to get to: 10000 curr count: 6126
2021/12/10 12:39:05 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 12:39:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:39:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:39:10 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 12:39:15 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 12:39:20 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 12:39:25 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 12:39:25 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/10 12:39:25 No eventing-con process running
2021/12/10 12:39:25 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/10 12:39:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:39:37 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 12:39:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:39:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:39:48 Waiting for dst bucket item count to get to: 0 curr count: 8614
2021/12/10 12:39:53 Waiting for dst bucket item count to get to: 0 curr count: 3614
2021/12/10 12:39:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (160.47s)
=== RUN   TestTimerOverwrite
2021/12/10 12:40:03 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 12:40:03 App: TestTimerOverwrite got un-deployed
2021/12/10 12:40:03 No eventing-con process running
2021/12/10 12:40:03 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/10 12:40:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:40:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:40:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:40:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:40:17 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:40:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/10 12:40:23 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 12:40:28 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 12:40:33 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 12:40:38 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 12:40:38 App: TestTimerOverwrite got deployed
2021/12/10 12:40:48 Starting bucket ops to source bucket
2021/12/10 12:40:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 12:41:57 Waiting for dst bucket item count to get to: 5000 curr count: 1217
2021/12/10 12:42:02 Waiting for dst bucket item count to get to: 5000 curr count: 3718
2021/12/10 12:42:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 12:42:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:42:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:42:17 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 12:42:22 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 12:42:27 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 12:42:32 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 12:42:32 App: TestTimerOverwrite got un-deployed
2021/12/10 12:42:32 No eventing-con process running
2021/12/10 12:42:32 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/10 12:42:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:42:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:42:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 12:42:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:42:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:43:00 Waiting for dst bucket item count to get to: 0 curr count: 3799
2021/12/10 12:43:05 Waiting for dst bucket item count to get to: 0 curr count: 3799
2021/12/10 12:43:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (191.89s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/10 12:43:10 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 12:43:11 Removing node(s): [] from the cluster
2021/12/10 12:43:16 Rebalance progress: 100
2021/12/10 12:43:16 Failing over: [127.0.0.1:19003]
2021/12/10 12:43:21 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/10 12:43:21 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/10 12:43:21 No eventing-con process running
2021/12/10 12:43:21 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/10 12:43:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:43:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:43:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:43:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:43:34 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:43:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 12:43:35 Starting bucket ops to source bucket
2021/12/10 12:43:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:43:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:43:45 Context: Post to set retry counter, response code: 200 dump: 
2021/12/10 12:43:45 set retry counter, response &{[] }
2021/12/10 12:44:50 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/10 12:44:50 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/10 12:44:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:44:50 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/10 12:44:55 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/10 12:44:55 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/10 12:44:55 No eventing-con process running
2021/12/10 12:44:55 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/10 12:44:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:45:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:45:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:45:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:45:18 Waiting for dst bucket item count to get to: 0 curr count: 1917
2021/12/10 12:45:23 Waiting for dst bucket item count to get to: 0 curr count: 251
2021/12/10 12:45:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:45:23 Removing node(s): [] from the cluster
2021/12/10 12:45:28 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (142.83s)
=== RUN   TestOnUpdateSrcMutation
2021/12/10 12:45:38 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 12:45:38 App: TestOnUpdateSrcMutation got un-deployed
2021/12/10 12:45:38 No eventing-con process running
2021/12/10 12:45:38 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/10 12:45:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:45:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:45:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:45:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:45:51 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:45:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/10 12:45:52 Starting bucket ops to source bucket
2021/12/10 12:46:14 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 12:46:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:46:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:46:19 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 12:46:24 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 12:46:29 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 12:46:34 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 12:46:34 App: TestOnUpdateSrcMutation got un-deployed
2021/12/10 12:46:35 No eventing-con process running
2021/12/10 12:46:35 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/10 12:46:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:46:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 12:46:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:46:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:46:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (84.35s)
=== RUN   TestOnDeleteSrcMutation
2021/12/10 12:47:02 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 12:47:03 App: TestOnDeleteSrcMutation got un-deployed
2021/12/10 12:47:03 No eventing-con process running
2021/12/10 12:47:03 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/10 12:47:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:47:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:47:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:47:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:47:16 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:47:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/10 12:47:17 Starting bucket ops to source bucket
2021/12/10 12:47:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 12:47:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:47:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:47:50 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 12:47:55 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 12:48:00 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 12:48:06 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 12:48:06 App: TestOnDeleteSrcMutation got un-deployed
2021/12/10 12:48:06 No eventing-con process running
2021/12/10 12:48:06 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/10 12:48:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:48:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:48:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:48:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:48:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (91.09s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/10 12:48:34 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 12:48:34 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/10 12:48:34 No eventing-con process running
2021/12/10 12:48:34 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/10 12:48:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:48:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:48:42 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:48:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:48:47 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:48:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 12:48:48 Starting bucket ops to source bucket
2021/12/10 12:49:25 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 12:49:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:49:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:49:30 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 12:49:35 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 12:49:40 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 12:49:45 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 12:49:45 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/10 12:49:45 No eventing-con process running
2021/12/10 12:49:45 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/10 12:49:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:49:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 12:49:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:49:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:50:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.31s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/10 12:50:13 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 12:50:13 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/10 12:50:13 No eventing-con process running
2021/12/10 12:50:13 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/10 12:50:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:50:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:50:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:50:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:50:26 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:50:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 12:50:27 Starting bucket ops to source bucket
2021/12/10 12:51:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 12:51:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 12:51:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:51:21 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 12:51:26 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 12:51:31 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 12:51:36 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 12:51:36 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/10 12:51:36 No eventing-con process running
2021/12/10 12:51:36 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/10 12:51:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:51:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 12:51:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:51:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:51:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (111.10s)
=== RUN   TestInterHandlerRecursion
2021/12/10 12:52:04 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 12:52:04 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 12:52:04 No eventing-con process running
2021/12/10 12:52:04 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/10 12:52:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:52:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:52:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:52:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:52:22 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 12:52:22 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 12:52:22 No eventing-con process running
2021/12/10 12:52:22 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/10 12:52:23 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:52:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:52:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:52:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:52:35 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:52:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/10 12:52:36 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/10 12:52:41 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 12:52:46 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 12:52:51 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 12:52:56 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 12:52:56 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/10 12:52:56 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:52:56 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/10 12:52:56 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/10 12:52:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:53:01 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 12:53:07 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 12:53:12 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 12:53:17 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 12:53:17 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 12:53:17 No eventing-con process running
2021/12/10 12:53:17 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/10 12:53:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:53:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:53:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:53:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:53:35 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 12:53:35 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 12:53:35 No eventing-con process running
2021/12/10 12:53:35 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/10 12:53:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:53:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:53:43 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:53:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.53s)
=== RUN   TestInterBucketRecursion
2021/12/10 12:53:53 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:53:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 12:53:54 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 12:53:54 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:53:54 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/10 12:53:59 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 12:54:04 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 12:54:09 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 12:54:14 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 12:54:14 App: TestInterBucketRecursion_function1 got deployed
2021/12/10 12:54:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:54:19 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 12:54:24 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 12:54:29 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 12:54:34 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 12:54:34 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/10 12:54:34 No eventing-con process running
2021/12/10 12:54:34 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/10 12:54:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:54:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:54:42 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:54:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:54:52 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/10 12:54:52 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/10 12:54:52 No eventing-con process running
2021/12/10 12:54:52 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/10 12:54:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:54:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:55:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:55:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.44s)
=== RUN   TestLargeHandler
2021/12/10 12:55:05 Changed force_compress value to false
2021/12/10 12:55:05 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:55:05 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/10 12:55:05 Changed force_compress value to true
2021/12/10 12:55:05 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:55:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/10 12:55:12 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 12:55:17 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 12:55:22 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 12:55:27 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 12:55:27 App: TestLargeHandler_function got deployed
2021/12/10 12:55:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:55:32 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 12:55:37 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 12:55:42 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 12:55:47 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 12:55:47 App: TestLargeHandler_function got un-deployed
2021/12/10 12:55:47 No eventing-con process running
2021/12/10 12:55:48 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/10 12:55:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:55:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:55:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:56:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.48s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/10 12:56:10 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 12:56:10 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 12:56:10 No eventing-con process running
2021/12/10 12:56:10 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/10 12:56:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:56:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:56:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:56:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:56:29 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 12:56:29 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 12:56:29 No eventing-con process running
2021/12/10 12:56:29 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/10 12:56:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:56:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:56:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:56:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:56:42 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:56:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/10 12:56:43 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/10 12:56:48 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 12:56:53 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 12:56:58 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 12:57:03 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 12:57:03 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/10 12:57:03 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:57:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/10 12:57:04 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/10 12:57:09 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 12:57:14 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 12:57:19 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 12:57:24 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 12:57:24 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/10 12:57:24 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/10 12:57:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:57:29 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 12:57:34 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 12:57:39 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 12:57:44 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 12:57:44 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 12:57:44 No eventing-con process running
2021/12/10 12:57:44 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/10 12:57:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:57:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:57:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:57:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:57:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:58:04 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 12:58:09 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 12:58:14 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 12:58:19 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 12:58:19 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 12:58:19 No eventing-con process running
2021/12/10 12:58:19 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/10 12:58:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:58:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:58:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:58:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.31s)
=== RUN   TestAllowInterBucketRecursion
2021/12/10 12:58:37 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:58:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 12:58:38 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 12:58:43 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 12:58:48 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 12:58:53 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 12:58:58 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 12:58:58 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/10 12:58:58 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 12:58:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/10 12:58:59 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/10 12:59:04 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 12:59:09 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 12:59:14 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 12:59:19 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 12:59:19 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/10 12:59:19 TestAllowInterBucketRecursion_function2 is deployed
2021/12/10 12:59:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:59:24 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 12:59:29 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 12:59:34 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 12:59:39 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 12:59:39 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/10 12:59:39 No eventing-con process running
2021/12/10 12:59:39 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/10 12:59:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:59:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:59:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 12:59:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 12:59:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 12:59:59 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 13:00:04 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 13:00:09 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 13:00:14 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 13:00:14 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/10 13:00:14 No eventing-con process running
2021/12/10 13:00:14 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/10 13:00:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:00:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:00:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:00:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.25s)
=== RUN   TestInterBucketRecursion2
2021/12/10 13:00:33 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:00:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:00:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:00:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:00:55 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:00:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 13:01:01 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 13:01:06 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 13:01:11 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 13:01:16 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 13:01:16 App: TestInterBucketRecursion2_function1 got deployed
2021/12/10 13:01:16 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:01:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/10 13:01:23 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 13:01:28 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 13:01:33 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 13:01:38 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 13:01:38 App: TestInterBucketRecursion2_function2 got deployed
2021/12/10 13:01:38 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:01:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/10 13:01:44 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 13:01:49 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 13:01:54 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 13:01:59 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 13:01:59 App: TestInterBucketRecursion2_function3 got deployed
2021/12/10 13:01:59 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:01:59 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/10 13:01:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:02:04 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 13:02:09 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 13:02:14 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 13:02:19 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 13:02:19 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/10 13:02:19 No eventing-con process running
2021/12/10 13:02:19 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/10 13:02:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:02:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:02:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:02:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:02:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:02:37 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 13:02:42 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 13:02:47 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 13:02:52 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 13:02:52 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/10 13:02:52 No eventing-con process running
2021/12/10 13:02:52 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/10 13:02:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:02:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:03:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:03:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:03:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:03:11 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 13:03:16 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 13:03:21 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 13:03:26 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 13:03:26 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/10 13:03:26 No eventing-con process running
2021/12/10 13:03:26 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/10 13:03:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:03:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:03:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:03:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:03:44 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/10 13:03:44 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/10 13:03:44 No eventing-con process running
2021/12/10 13:03:44 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/10 13:03:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:03:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:03:52 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:03:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (211.84s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/10 13:04:04 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 13:04:09 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/10 13:04:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:04:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:04:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:04:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:05:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:05:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:05:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:05:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:06:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:06:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:06:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:06:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:07:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:07:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:07:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:07:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:08:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:08:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:08:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:08:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:09:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:09:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:09:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 13:09:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:44} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/10 13:10:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:10:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 13:10:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 13:10:17 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/10 13:10:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:10:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 13:10:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 13:10:23 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/10 13:10:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:10:28 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/10 13:10:28 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/10 13:10:28 eventingSetup.go:46: No eventing-con process running
2021/12/10 13:10:28 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/10 13:10:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:10:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:10:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:10:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:10:46 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:10:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 13:10:52 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 13:10:57 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 13:11:02 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 13:11:07 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 13:11:07 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: bdaa326d7c24af83337ce72a470d792f
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: ae5c12a6d0163652e7b530be4e677ac7
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 8412b11f9f4b861c67c28e98cbac77fc
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/10 13:11:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:11:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 13:11:12 Rebalance progress: 29.16666666666666
2021/12/10 13:11:17 Rebalance progress: 29.16666666666666
2021/12/10 13:11:22 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: bdaa326d7c24af83337ce72a470d792f
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: ae5c12a6d0163652e7b530be4e677ac7
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
Producer node: 127.0.0.1:9303	Node UUID: 8412b11f9f4b861c67c28e98cbac77fc
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/10 13:11:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:11:22 handler_test.go:1674: Deleting source bucket: default
2021/12/10 13:11:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 13:11:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 13:11:29 Rebalance progress: 29.16666666666666
2021/12/10 13:11:34 Rebalance progress: 29.16666666666666
2021/12/10 13:11:39 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: stopped
	len: 122
	vb list dump: [2, 5, 15, 23, 38, 41, 54, 57-59, 76, 90, 94, 97, 115, 129, 134, 138-139, 146, 150, 171, 174, 179, 200-201, 212, 214, 225, 248, 259, 263, 268, 281, 287, 289, 297, 323-324, 327, 329, 335, 343, 349, 358, 362, 371, 380-381, 393-394, 396, 399, 419, 428, 430, 436, 457-461, 470, 476, 478, 483, 486, 491-492, 501, 570, 572-573, 580, 590, 595, 620, 626, 629, 647, 678-679, 691, 694, 703, 707, 716, 722, 729, 739, 755, 762, 764, 767-768, 772, 792, 800, 810, 822, 840-841, 844, 847, 854, 877, 883, 886, 897, 899, 954, 962, 965, 972, 976, 978-979, 997, 1007, 1013, 1015, 1020]
	stream status: running
	len: 902
	vb list dump: [0-1, 3-4, 6-14, 16-22, 24-37, 39-40, 42-53, 55-56, 60-75, 77-89, 91-93, 95-96, 98-114, 116-128, 130-133, 135-137, 140-145, 147-149, 151-170, 172-173, 175-178, 180-199, 202-211, 213, 215-224, 226-247, 249-258, 260-262, 264-267, 269-280, 282-286, 288, 290-296, 298-322, 325-326, 328, 330-334, 336-342, 344-348, 350-357, 359-361, 363-370, 372-379, 382-392, 395, 397-398, 400-418, 420-427, 429, 431-435, 437-456, 462-469, 471-475, 477, 479-482, 484-485, 487-490, 493-500, 502-569, 571, 574-579, 581-589, 591-594, 596-619, 621-625, 627-628, 630-646, 648-677, 680-690, 692-693, 695-702, 704-706, 708-715, 717-721, 723-728, 730-738, 740-754, 756-761, 763, 765-766, 769-771, 773-791, 793-799, 801-809, 811-821, 823-839, 842-843, 845-846, 848-853, 855-876, 878-882, 884-885, 887-896, 898, 900-953, 955-961, 963-964, 966-971, 973-975, 977, 980-996, 998-1006, 1008-1012, 1014, 1016-1019, 1021-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: 8412b11f9f4b861c67c28e98cbac77fc
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [456, 462-469, 471-475, 477, 479-482, 484-485, 487-490, 493-500, 502-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [571, 574-579, 581-589, 591-594, 596-619, 621-625, 627-628, 630-646, 648-677, 680-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [342, 344-348, 350-357, 359-361, 363-370, 372-379, 382-392, 395, 397-398, 400-418, 420-427, 429, 431-435, 437-455]
Producer node: 127.0.0.1:9302	Node UUID: bdaa326d7c24af83337ce72a470d792f
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [0-1, 3-4, 6-14, 16-22, 24-37, 39-40, 42-53, 55-56, 60-75, 77-89, 91-93, 95-96, 98-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [114, 116-128, 130-133, 135-137, 140-145, 147-149, 151-170, 172-173, 175-178, 180-199, 202-211, 213, 215-224, 226-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-247, 249-258, 260-262, 264-267, 269-280, 282-286, 288, 290-296, 298-322, 325-326, 328, 330-334, 336-341]
Producer node: 172.31.2.134:9300	Node UUID: ae5c12a6d0163652e7b530be4e677ac7
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-799, 801-809, 811-821, 823-839, 842-843, 845-846, 848-853, 855-876, 878-882, 884-885, 887-896, 898, 900-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-953, 955-961, 963-964, 966-971, 973-975, 977, 980-996, 998-1006, 1008-1012, 1014, 1016-1019, 1021-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [683-690, 692-693, 695-702, 704-706, 708-715, 717-721, 723-728, 730-738, 740-754, 756-761, 763, 765-766, 769-771, 773-791, 793-796]


2021/12/10 13:11:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:11:44 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/10 13:11:44 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/10 13:11:49 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/10 13:11:49 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/10 13:11:49 eventingSetup.go:46: No eventing-con process running
2021/12/10 13:11:49 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/10 13:11:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:11:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:11:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:12:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:12:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/10 13:12:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 13:12:08 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (491.66s)
=== RUN   TestN1QLRecursion
2021/12/10 13:12:11 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 13:12:11 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/10 13:12:11 Removing node(s): [] from the cluster
2021/12/10 13:12:16 Rebalance progress: 100
2021/12/10 13:12:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:12:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:12:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:12:39 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:12:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 13:12:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 13:12:45 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 13:12:50 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 13:12:55 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 13:13:00 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 13:13:01 App: TestN1QLRecursion_function1 got deployed
2021/12/10 13:13:01 TestN1QLRecursion_function1 is deployed
2021/12/10 13:13:01 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:13:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/10 13:13:02 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/10 13:13:07 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 13:13:12 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 13:13:17 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 13:13:22 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 13:13:22 App: TestN1QLRecursion_function3 got deployed
2021/12/10 13:13:22 TestN1QLRecursion_function3 is deployed
2021/12/10 13:13:22 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:13:22 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/10 13:13:22 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/10 13:13:22 Success: TestN1QLRecursion_function2 is not deployed
2021/12/10 13:13:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:13:27 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 13:13:32 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 13:13:37 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 13:13:42 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 13:13:42 App: TestN1QLRecursion_function1 got un-deployed
2021/12/10 13:13:42 No eventing-con process running
2021/12/10 13:13:42 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/10 13:13:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:13:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:13:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:13:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:13:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:14:01 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 13:14:06 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 13:14:11 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 13:14:16 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 13:14:16 App: TestN1QLRecursion_function3 got un-deployed
2021/12/10 13:14:16 No eventing-con process running
2021/12/10 13:14:16 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/10 13:14:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:14:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:14:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:14:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:14:31 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 13:14:36 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.34s)
=== RUN   TestN1QLAllowRecursion
2021/12/10 13:14:36 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 13:14:36 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/10 13:14:36 Removing node(s): [] from the cluster
2021/12/10 13:14:41 Rebalance progress: 100
2021/12/10 13:14:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:14:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:14:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:15:03 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:15:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 13:15:05 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 13:15:10 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 13:15:15 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 13:15:20 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 13:15:25 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 13:15:25 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/10 13:15:25 TestN1QLAllowRecursion_function1 is deployed
2021/12/10 13:15:25 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:15:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/10 13:15:26 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/10 13:15:31 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 13:15:36 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 13:15:41 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 13:15:46 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 13:15:46 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/10 13:15:46 TestN1QLAllowRecursion_function3 is deployed
2021/12/10 13:15:46 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:15:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/10 13:15:47 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/10 13:15:52 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 13:15:57 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 13:16:02 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 13:16:07 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 13:16:07 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/10 13:16:07 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/10 13:16:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:16:12 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 13:16:17 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 13:16:22 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 13:16:27 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 13:16:27 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/10 13:16:27 No eventing-con process running
2021/12/10 13:16:27 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/10 13:16:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:16:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:16:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:16:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:16:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:16:46 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 13:16:51 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 13:16:56 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 13:17:01 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 13:17:01 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/10 13:17:01 No eventing-con process running
2021/12/10 13:17:01 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/10 13:17:03 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:17:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:17:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:17:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:17:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:17:20 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 13:17:25 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 13:17:30 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 13:17:35 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 13:17:35 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/10 13:17:35 No eventing-con process running
2021/12/10 13:17:35 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/10 13:17:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:17:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:17:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:17:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:17:50 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 13:17:55 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (199.56s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/10 13:17:55 Sleeping for some time
2021/12/10 13:18:05 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 13:18:05 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/10 13:18:05 No eventing-con process running
2021/12/10 13:18:05 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/10 13:18:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:18:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:18:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:18:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:18:19 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:18:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 13:18:20 Starting bucket ops to source bucket
2021/12/10 13:19:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 13:19:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 13:19:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 13:19:14 Starting bucket ops to source bucket
2021/12/10 13:19:15 Starting bucket ops to source bucket
2021/12/10 13:20:21 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/10 13:20:21 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/10 13:20:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:20:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:20:26 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 13:20:31 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 13:20:36 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 13:20:41 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 13:20:41 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/10 13:20:41 No eventing-con process running
2021/12/10 13:20:41 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/10 13:20:43 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:20:53 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 13:20:58 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/10 13:20:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:21:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:21:10 Waiting for dst bucket item count to get to: 0 curr count: 4623
2021/12/10 13:21:15 Waiting for dst bucket item count to get to: 0 curr count: 4623
2021/12/10 13:21:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.64s)
=== RUN   TestTimerOverWriteSameReference
2021/12/10 13:21:20 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 13:21:20 App: TestTimerOverWriteSameReference got un-deployed
2021/12/10 13:21:20 No eventing-con process running
2021/12/10 13:21:20 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/10 13:21:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:21:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:21:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:21:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:21:34 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:21:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/10 13:21:40 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 13:21:45 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 13:21:50 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 13:21:55 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 13:21:55 App: TestTimerOverWriteSameReference got deployed
2021/12/10 13:22:05 Starting bucket ops to source bucket
2021/12/10 13:23:06 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/10 13:23:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:23:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:23:21 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 13:23:26 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 13:23:31 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 13:23:36 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 13:23:36 App: TestTimerOverWriteSameReference got un-deployed
2021/12/10 13:23:36 No eventing-con process running
2021/12/10 13:23:36 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/10 13:23:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:23:48 Waiting for dst bucket item count to get to: 0 curr count: 644
2021/12/10 13:23:53 Waiting for dst bucket item count to get to: 0 curr count: 144
2021/12/10 13:23:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:23:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:24:05 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/10 13:24:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.73s)
=== RUN   TestCancelTimerBucketop
2021/12/10 13:24:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 13:24:10 App: TestCancelTimerBucketop got un-deployed
2021/12/10 13:24:10 No eventing-con process running
2021/12/10 13:24:10 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/10 13:24:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:24:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:24:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:24:23 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:24:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/10 13:24:30 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 13:24:35 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 13:24:40 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 13:24:45 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 13:24:45 App: TestCancelTimerBucketop got deployed
2021/12/10 13:24:45 Starting bucket ops to source bucket
2021/12/10 13:24:52 Starting bucket ops to source bucket
2021/12/10 13:25:54 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 13:25:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:25:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:25:59 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 13:26:04 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 13:26:09 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 13:26:14 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 13:26:14 App: TestCancelTimerBucketop got un-deployed
2021/12/10 13:26:14 No eventing-con process running
2021/12/10 13:26:14 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/10 13:26:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:26:26 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 13:26:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:26:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:26:38 Waiting for dst bucket item count to get to: 0 curr count: 636
2021/12/10 13:26:43 Waiting for dst bucket item count to get to: 0 curr count: 136
2021/12/10 13:26:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.10s)
=== RUN   TestJSExpiryDate
2021/12/10 13:26:48 Starting bucket ops to source bucket
2021/12/10 13:27:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 13:27:06 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/10 13:27:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/10 13:27:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:27:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:27:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:27:42 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 13:27:47 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 13:27:52 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 13:27:57 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 13:27:57 App: TestJSExpiryDate got un-deployed
2021/12/10 13:27:57 No eventing-con process running
2021/12/10 13:27:57 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/10 13:27:59 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:28:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:28:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:28:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (88.63s)
=== RUN   TestBinaryDoc
2021/12/10 13:28:17 Starting bucket ops to source bucket
2021/12/10 13:28:17 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:28:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/10 13:28:18 Starting bucket ops to source bucket
2021/12/10 13:28:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:28:18 Deleting source bucket docs now
2021/12/10 13:28:18 Starting bucket ops to source bucket
2021/12/10 13:28:55 Waiting for dst bucket item count to get to: 32 curr count: 20
2021/12/10 13:29:00 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/10 13:29:05 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/10 13:29:05 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/10 13:29:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:29:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:29:10 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 13:29:15 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 13:29:20 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 13:29:25 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 13:29:25 App: TestBinaryDoc got un-deployed
2021/12/10 13:29:25 No eventing-con process running
2021/12/10 13:29:25 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/10 13:29:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:29:37 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/10 13:29:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:29:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:29:49 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/10 13:29:54 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/10 13:29:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (102.48s)
=== RUN   TestConstantBindings
2021/12/10 13:29:54 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:29:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/10 13:29:55 Starting bucket ops to source bucket
2021/12/10 13:30:01 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/10 13:30:06 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/10 13:30:11 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/10 13:30:16 Waiting for dst bucket item count to get to: 6000 curr count: 3992
2021/12/10 13:30:21 Waiting for dst bucket item count to get to: 6000 curr count: 3992
2021/12/10 13:30:21 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/10 13:30:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:30:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:30:26 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 13:30:31 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 13:30:36 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 13:30:41 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 13:30:41 App: TestConstantBindings got un-deployed
2021/12/10 13:30:41 No eventing-con process running
2021/12/10 13:30:41 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/10 13:30:43 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:30:53 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 13:30:58 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 13:30:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:31:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:31:10 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/10 13:31:15 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/10 13:31:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.62s)
=== RUN   TestCodeInsight
2021/12/10 13:31:20 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 13:31:20 App: TestCodeInsight got un-deployed
2021/12/10 13:31:20 No eventing-con process running
2021/12/10 13:31:20 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/10 13:31:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:31:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:31:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:31:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:31:33 Starting bucket ops to source bucket
2021/12/10 13:31:34 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:31:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/10 13:31:40 Waiting for app: TestCodeInsight to get deployed
2021/12/10 13:31:45 Waiting for app: TestCodeInsight to get deployed
2021/12/10 13:31:50 Waiting for app: TestCodeInsight to get deployed
2021/12/10 13:31:55 Waiting for app: TestCodeInsight to get deployed
2021/12/10 13:31:55 App: TestCodeInsight got deployed
2021/12/10 13:32:00 Waiting for dst bucket item count to get to: 1000 curr count: 535
2021/12/10 13:32:00 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 13:32:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:32:05 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 13:32:10 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 13:32:15 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 13:32:20 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 13:32:20 App: TestCodeInsight got un-deployed
2021/12/10 13:32:20 No eventing-con process running
2021/12/10 13:32:20 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/10 13:32:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:32:32 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 13:32:37 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 13:32:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:32:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:32:49 Waiting for dst bucket item count to get to: 0 curr count: 984
2021/12/10 13:32:54 Waiting for dst bucket item count to get to: 0 curr count: 484
2021/12/10 13:32:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.28s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.06s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12896.171s

Functional tests took 3 hours, 35 minutes and 0 seconds.

N1ql tests

2021/12/10 13:33:05 Started n1ql
2021/12/10 13:33:14 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/10 13:33:15 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/10 13:33:15 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 13:33:15 Removing node(s): [] from the cluster
2021/12/10 13:33:20 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/10 13:33:25 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/10 13:33:25 App: TestRecursiveMutationN1QL got un-deployed
2021/12/10 13:33:25 No eventing-con process running
2021/12/10 13:33:25 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/10 13:33:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:33:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:33:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:33:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:33:39 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:33:39 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.44s)
=== RUN   TestFlexReset1
2021/12/10 13:33:44 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 13:33:44 App: TestFlexReset1 got un-deployed
2021/12/10 13:33:44 No eventing-con process running
2021/12/10 13:33:44 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/10 13:33:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:33:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:33:52 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:33:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:33:57 Starting bucket ops to source bucket
2021/12/10 13:33:58 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:33:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/10 13:34:04 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 13:34:09 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 13:34:14 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 13:34:19 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 13:34:19 App: TestFlexReset1 got deployed
2021/12/10 13:34:24 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/10 13:34:24 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 13:34:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:34:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:34:29 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 13:34:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 13:34:39 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 13:34:44 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 13:34:44 App: TestFlexReset1 got un-deployed
2021/12/10 13:34:44 No eventing-con process running
2021/12/10 13:34:44 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/10 13:34:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:34:56 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 13:34:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:34:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:35:08 Waiting for dst bucket item count to get to: 0 curr count: 643
2021/12/10 13:35:13 Waiting for dst bucket item count to get to: 0 curr count: 143
2021/12/10 13:35:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.31s)
=== RUN   TestFlexReset2
2021/12/10 13:35:18 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 13:35:18 App: TestFlexReset2 got un-deployed
2021/12/10 13:35:18 No eventing-con process running
2021/12/10 13:35:18 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/10 13:35:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:35:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:35:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:35:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:35:31 Starting bucket ops to source bucket
2021/12/10 13:35:32 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:35:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/10 13:35:38 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 13:35:43 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 13:35:48 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 13:35:53 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 13:35:53 App: TestFlexReset2 got deployed
2021/12/10 13:35:58 Waiting for dst bucket item count to get to: 1000 curr count: 789
2021/12/10 13:36:03 Waiting for dst bucket item count to get to: 1000 curr count: 789
2021/12/10 13:36:03 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 13:36:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:36:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:36:09 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 13:36:14 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 13:36:19 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 13:36:24 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 13:36:24 App: TestFlexReset2 got un-deployed
2021/12/10 13:36:24 No eventing-con process running
2021/12/10 13:36:24 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/10 13:36:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:36:35 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 13:36:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:36:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:36:47 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 13:36:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (94.23s)
=== RUN   TestN1QLLabelledBreak
2021/12/10 13:36:52 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 13:36:52 App: TestN1QLLabelledBreak got un-deployed
2021/12/10 13:36:52 No eventing-con process running
2021/12/10 13:36:52 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/10 13:36:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:36:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:37:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:37:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:37:06 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:37:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/10 13:37:12 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 13:37:17 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 13:37:22 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 13:37:27 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 13:37:27 App: TestN1QLLabelledBreak got deployed
2021/12/10 13:37:27 Starting bucket ops to source bucket
2021/12/10 13:37:36 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/10 13:37:41 Waiting for dst bucket item count to get to: 10000 curr count: 170
2021/12/10 13:37:46 Waiting for dst bucket item count to get to: 10000 curr count: 170
2021/12/10 13:37:51 Waiting for dst bucket item count to get to: 10000 curr count: 3434
2021/12/10 13:37:56 Waiting for dst bucket item count to get to: 10000 curr count: 3434
2021/12/10 13:38:01 Waiting for dst bucket item count to get to: 10000 curr count: 7401
2021/12/10 13:38:06 Waiting for dst bucket item count to get to: 10000 curr count: 7401
2021/12/10 13:38:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 13:38:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:38:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:38:11 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 13:38:16 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 13:38:21 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 13:38:26 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 13:38:26 App: TestN1QLLabelledBreak got un-deployed
2021/12/10 13:38:26 No eventing-con process running
2021/12/10 13:38:27 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/10 13:38:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:38:38 Waiting for dst bucket item count to get to: 0 curr count: 3733
2021/12/10 13:38:43 Waiting for dst bucket item count to get to: 0 curr count: 1233
2021/12/10 13:38:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:38:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:38:55 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 13:38:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (127.96s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/10 13:39:00 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 13:39:00 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/10 13:39:00 No eventing-con process running
2021/12/10 13:39:00 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/10 13:39:02 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:39:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:39:09 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:39:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:39:14 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:39:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/10 13:39:20 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 13:39:25 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 13:39:30 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 13:39:35 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 13:39:35 App: TestN1QLUnlabelledBreak got deployed
2021/12/10 13:39:35 Starting bucket ops to source bucket
2021/12/10 13:39:44 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/10 13:39:49 Waiting for dst bucket item count to get to: 10000 curr count: 255
2021/12/10 13:39:54 Waiting for dst bucket item count to get to: 10000 curr count: 746
2021/12/10 13:39:59 Waiting for dst bucket item count to get to: 10000 curr count: 3264
2021/12/10 13:40:04 Waiting for dst bucket item count to get to: 10000 curr count: 6169
2021/12/10 13:40:04 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 13:40:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:40:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:40:09 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 13:40:14 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 13:40:19 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 13:40:24 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 13:40:24 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/10 13:40:24 No eventing-con process running
2021/12/10 13:40:25 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/10 13:40:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:40:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 13:40:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:40:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:40:48 Waiting for dst bucket item count to get to: 0 curr count: 7063
2021/12/10 13:40:53 Waiting for dst bucket item count to get to: 0 curr count: 2063
2021/12/10 13:40:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (117.99s)
=== RUN   TestN1QLThrowStatement
2021/12/10 13:40:58 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 13:40:58 App: TestN1QLThrowStatement got un-deployed
2021/12/10 13:40:58 No eventing-con process running
2021/12/10 13:40:58 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/10 13:41:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:41:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:41:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:41:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:41:12 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:41:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/10 13:41:18 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 13:41:23 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 13:41:28 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 13:41:33 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 13:41:33 App: TestN1QLThrowStatement got deployed
2021/12/10 13:41:33 Starting bucket ops to source bucket
2021/12/10 13:41:42 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/10 13:41:47 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/10 13:41:52 Waiting for dst bucket item count to get to: 10000 curr count: 1054
2021/12/10 13:41:57 Waiting for dst bucket item count to get to: 10000 curr count: 1054
2021/12/10 13:42:02 Waiting for dst bucket item count to get to: 10000 curr count: 5129
2021/12/10 13:42:07 Waiting for dst bucket item count to get to: 10000 curr count: 7152
2021/12/10 13:42:12 Waiting for dst bucket item count to get to: 10000 curr count: 9152
2021/12/10 13:42:17 Waiting for dst bucket item count to get to: 10000 curr count: 9785
2021/12/10 13:42:17 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 13:42:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:42:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:42:22 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 13:42:27 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 13:42:32 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 13:42:37 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 13:42:37 App: TestN1QLThrowStatement got un-deployed
2021/12/10 13:42:37 No eventing-con process running
2021/12/10 13:42:37 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/10 13:42:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:42:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 13:42:54 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 13:42:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:42:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:43:06 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 13:43:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (132.90s)
=== RUN   TestN1QLNestedForLoop
2021/12/10 13:43:11 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 13:43:11 App: TestN1QLNestedForLoop got un-deployed
2021/12/10 13:43:11 No eventing-con process running
2021/12/10 13:43:11 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/10 13:43:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:43:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:43:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:43:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:43:25 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:43:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/10 13:43:31 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 13:43:36 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 13:43:41 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 13:43:46 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 13:43:46 App: TestN1QLNestedForLoop got deployed
2021/12/10 13:43:46 Starting bucket ops to source bucket
2021/12/10 13:43:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 13:44:00 Waiting for dst bucket item count to get to: 5000 curr count: 162
2021/12/10 13:44:05 Waiting for dst bucket item count to get to: 5000 curr count: 162
2021/12/10 13:44:10 Waiting for dst bucket item count to get to: 5000 curr count: 2255
2021/12/10 13:44:15 Waiting for dst bucket item count to get to: 5000 curr count: 2255
2021/12/10 13:44:20 Waiting for dst bucket item count to get to: 5000 curr count: 4682
2021/12/10 13:44:25 Waiting for dst bucket item count to get to: 5000 curr count: 4682
2021/12/10 13:44:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 13:44:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:44:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:44:30 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 13:44:35 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 13:44:40 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 13:44:45 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 13:44:45 App: TestN1QLNestedForLoop got un-deployed
2021/12/10 13:44:45 No eventing-con process running
2021/12/10 13:44:45 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/10 13:44:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:44:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 13:44:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:44:59 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:45:09 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 13:45:14 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 13:45:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.01s)
=== RUN   TestN1QLAutoClose
2021/12/10 13:45:19 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 13:45:19 App: TestN1QLAutoClose got un-deployed
2021/12/10 13:45:19 No eventing-con process running
2021/12/10 13:45:19 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/10 13:45:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:45:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:45:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:45:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:45:33 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:45:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/10 13:45:39 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 13:45:44 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 13:45:49 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 13:45:54 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 13:45:54 App: TestN1QLAutoClose got deployed
2021/12/10 13:45:54 Starting bucket ops to source bucket
2021/12/10 13:45:59 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/10 13:46:04 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/10 13:46:09 Waiting for dst bucket item count to get to: 500 curr count: 9
2021/12/10 13:46:14 Waiting for dst bucket item count to get to: 500 curr count: 18
2021/12/10 13:46:19 Waiting for dst bucket item count to get to: 500 curr count: 30
2021/12/10 13:46:24 Waiting for dst bucket item count to get to: 500 curr count: 41
2021/12/10 13:46:29 Waiting for dst bucket item count to get to: 500 curr count: 52
2021/12/10 13:46:34 Waiting for dst bucket item count to get to: 500 curr count: 62
2021/12/10 13:46:39 Waiting for dst bucket item count to get to: 500 curr count: 74
2021/12/10 13:46:44 Waiting for dst bucket item count to get to: 500 curr count: 85
2021/12/10 13:46:49 Waiting for dst bucket item count to get to: 500 curr count: 95
2021/12/10 13:46:54 Waiting for dst bucket item count to get to: 500 curr count: 103
2021/12/10 13:46:59 Waiting for dst bucket item count to get to: 500 curr count: 103
2021/12/10 13:47:04 Waiting for dst bucket item count to get to: 500 curr count: 115
2021/12/10 13:47:09 Waiting for dst bucket item count to get to: 500 curr count: 125
2021/12/10 13:47:14 Waiting for dst bucket item count to get to: 500 curr count: 136
2021/12/10 13:47:20 Waiting for dst bucket item count to get to: 500 curr count: 159
2021/12/10 13:47:25 Waiting for dst bucket item count to get to: 500 curr count: 159
2021/12/10 13:47:30 Waiting for dst bucket item count to get to: 500 curr count: 181
2021/12/10 13:47:35 Waiting for dst bucket item count to get to: 500 curr count: 181
2021/12/10 13:47:40 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/10 13:47:45 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/10 13:47:50 Waiting for dst bucket item count to get to: 500 curr count: 226
2021/12/10 13:47:55 Waiting for dst bucket item count to get to: 500 curr count: 226
2021/12/10 13:48:00 Waiting for dst bucket item count to get to: 500 curr count: 246
2021/12/10 13:48:05 Waiting for dst bucket item count to get to: 500 curr count: 246
2021/12/10 13:48:10 Waiting for dst bucket item count to get to: 500 curr count: 268
2021/12/10 13:48:15 Waiting for dst bucket item count to get to: 500 curr count: 268
2021/12/10 13:48:20 Waiting for dst bucket item count to get to: 500 curr count: 289
2021/12/10 13:48:25 Waiting for dst bucket item count to get to: 500 curr count: 289
2021/12/10 13:48:30 Waiting for dst bucket item count to get to: 500 curr count: 311
2021/12/10 13:48:35 Waiting for dst bucket item count to get to: 500 curr count: 311
2021/12/10 13:48:40 Waiting for dst bucket item count to get to: 500 curr count: 334
2021/12/10 13:48:45 Waiting for dst bucket item count to get to: 500 curr count: 334
2021/12/10 13:48:50 Waiting for dst bucket item count to get to: 500 curr count: 355
2021/12/10 13:48:55 Waiting for dst bucket item count to get to: 500 curr count: 355
2021/12/10 13:49:00 Waiting for dst bucket item count to get to: 500 curr count: 376
2021/12/10 13:49:05 Waiting for dst bucket item count to get to: 500 curr count: 376
2021/12/10 13:49:10 Waiting for dst bucket item count to get to: 500 curr count: 398
2021/12/10 13:49:15 Waiting for dst bucket item count to get to: 500 curr count: 398
2021/12/10 13:49:20 Waiting for dst bucket item count to get to: 500 curr count: 419
2021/12/10 13:49:25 Waiting for dst bucket item count to get to: 500 curr count: 419
2021/12/10 13:49:30 Waiting for dst bucket item count to get to: 500 curr count: 441
2021/12/10 13:49:35 Waiting for dst bucket item count to get to: 500 curr count: 453
2021/12/10 13:49:40 Waiting for dst bucket item count to get to: 500 curr count: 464
2021/12/10 13:49:45 Waiting for dst bucket item count to get to: 500 curr count: 474
2021/12/10 13:49:50 Waiting for dst bucket item count to get to: 500 curr count: 486
2021/12/10 13:49:55 Waiting for dst bucket item count to get to: 500 curr count: 495
2021/12/10 13:49:55 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/10 13:49:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:49:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:50:00 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 13:50:05 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 13:50:10 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 13:50:15 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 13:50:15 App: TestN1QLAutoClose got un-deployed
2021/12/10 13:50:15 No eventing-con process running
2021/12/10 13:50:15 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/10 13:50:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:50:27 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 13:50:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:50:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:50:38 Waiting for dst bucket item count to get to: 0 curr count: 431
2021/12/10 13:50:43 Waiting for dst bucket item count to get to: 0 curr count: 181
2021/12/10 13:50:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.51s)
=== RUN   TestN1QLPosParams
2021/12/10 13:50:49 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 13:50:49 App: TestN1QLPosParams got un-deployed
2021/12/10 13:50:49 No eventing-con process running
2021/12/10 13:50:49 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/10 13:50:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:50:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:50:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:51:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:51:02 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:51:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/10 13:51:08 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 13:51:13 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 13:51:18 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 13:51:23 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 13:51:23 App: TestN1QLPosParams got deployed
2021/12/10 13:51:23 Starting bucket ops to source bucket
2021/12/10 13:51:33 Waiting for dst bucket item count to get to: 5000 curr count: 648
2021/12/10 13:51:38 Waiting for dst bucket item count to get to: 5000 curr count: 1958
2021/12/10 13:51:43 Waiting for dst bucket item count to get to: 5000 curr count: 3810
2021/12/10 13:51:48 Waiting for dst bucket item count to get to: 5000 curr count: 3810
2021/12/10 13:51:48 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 13:51:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:51:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:51:53 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 13:51:58 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 13:52:03 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 13:52:08 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 13:52:08 App: TestN1QLPosParams got un-deployed
2021/12/10 13:52:08 No eventing-con process running
2021/12/10 13:52:08 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/10 13:52:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:52:20 Waiting for dst bucket item count to get to: 0 curr count: 3114
2021/12/10 13:52:25 Waiting for dst bucket item count to get to: 0 curr count: 3114
2021/12/10 13:52:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:52:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:52:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 13:52:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (113.20s)
=== RUN   TestN1QLExhaustConnPool
2021/12/10 13:52:42 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 13:52:42 App: TestN1QLExhaustConnPool got un-deployed
2021/12/10 13:52:42 No eventing-con process running
2021/12/10 13:52:42 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/10 13:52:43 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:52:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:52:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:52:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:52:55 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:52:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/10 13:53:01 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 13:53:06 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 13:53:11 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 13:53:16 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 13:53:16 App: TestN1QLExhaustConnPool got deployed
2021/12/10 13:53:16 Starting bucket ops to source bucket
2021/12/10 13:53:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 13:53:27 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 13:53:32 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 13:53:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 13:53:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:53:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:53:37 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 13:53:42 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 13:53:47 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 13:53:52 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 13:53:52 App: TestN1QLExhaustConnPool got un-deployed
2021/12/10 13:53:52 No eventing-con process running
2021/12/10 13:53:52 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/10 13:53:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:54:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 13:54:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 13:54:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:54:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:54:20 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/10 13:54:25 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/10 13:54:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.70s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/10 13:54:30 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 13:54:30 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/10 13:54:30 No eventing-con process running
2021/12/10 13:54:30 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/10 13:54:32 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:54:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:54:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:54:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:54:44 Starting bucket ops to source bucket
2021/12/10 13:55:01 Starting bucket ops to source bucket
2021/12/10 13:55:19 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 12416
2021/12/10 13:55:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/10 13:55:25 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 13:55:30 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 13:55:35 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 13:55:40 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 13:55:40 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/10 13:55:40 Starting bucket ops to source bucket
2021/12/10 13:55:45 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 13:55:50 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 13:55:55 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 13:56:00 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 13:56:05 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 13:56:10 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 13:56:15 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 13:56:20 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 13:56:25 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 13:56:30 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 13:56:35 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 13:56:35 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/10 13:56:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:56:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:56:40 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 13:56:45 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 13:56:50 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 13:56:55 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 13:56:55 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/10 13:56:55 No eventing-con process running
2021/12/10 13:56:55 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/10 13:56:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:57:07 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/10 13:57:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:57:09 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:57:19 Waiting for dst bucket item count to get to: 0 curr count: 27604
2021/12/10 13:57:24 Waiting for dst bucket item count to get to: 0 curr count: 12604
2021/12/10 13:57:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (178.28s)
=== RUN   TestN1QLQueryConsistency
2021/12/10 13:57:29 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 13:57:29 App: n1qlquery_consistency got un-deployed
2021/12/10 13:57:29 No eventing-con process running
2021/12/10 13:57:29 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/10 13:57:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:57:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:57:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:57:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:57:42 Starting bucket ops to source bucket
2021/12/10 13:57:45 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/10 13:57:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/10 13:57:51 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 13:57:57 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 13:58:02 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 13:58:07 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 13:58:07 App: n1qlquery_consistency got deployed
2021/12/10 13:58:12 Waiting for dst bucket item count to get to: 5000 curr count: 155
2021/12/10 13:58:17 Waiting for dst bucket item count to get to: 5000 curr count: 2655
2021/12/10 13:58:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 13:58:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 13:58:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 13:58:22 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 13:58:27 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 13:58:32 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 13:58:37 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 13:58:37 App: n1qlquery_consistency got un-deployed
2021/12/10 13:58:37 No eventing-con process running
2021/12/10 13:58:37 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/10 13:58:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:58:49 Waiting for dst bucket item count to get to: 0 curr count: 4360
2021/12/10 13:58:54 Waiting for dst bucket item count to get to: 0 curr count: 1860
2021/12/10 13:58:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 13:58:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 13:59:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 13:59:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.59s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1556.361s

N1ql tests took 0 hours, 26 minutes and 0 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/10 14:00:41 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:59630->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:59630->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/10 14:00:59 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/10 14:01:04 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 14:01:04 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/10 14:01:04 No eventing-con process running
2021/12/10 14:01:04 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/10 14:01:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:01:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:01:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:01:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:01:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:01:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:01:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:01:23 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:01:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 14:01:29 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 14:01:34 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 14:01:39 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 14:01:44 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 14:01:44 App: TestDuplicateWithEventingRebIn got deployed
2021/12/10 14:01:44 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:01:44 Starting bucket ops to source bucket
2021/12/10 14:01:58 Removing node(s): [] from the cluster
2021/12/10 14:02:03 Rebalance progress: 0
2021/12/10 14:02:08 Rebalance progress: 0
2021/12/10 14:02:13 Rebalance progress: 25
2021/12/10 14:02:18 Rebalance progress: 25
2021/12/10 14:02:23 Rebalance progress: 25
2021/12/10 14:02:28 Rebalance progress: 26.29310344827586
2021/12/10 14:02:33 Rebalance progress: 98.70689655172413
2021/12/10 14:02:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:02:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:02:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:02:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 14:02:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:02:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:02:43 Rebalance progress: 25
2021/12/10 14:02:48 Rebalance progress: 25
2021/12/10 14:02:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:02:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:02:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:02:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 14:02:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:02:53 Waiting for src bucket item count to get to: 200000 curr count: 92464
2021/12/10 14:02:58 Waiting for src bucket item count to get to: 200000 curr count: 109095
2021/12/10 14:03:03 Waiting for src bucket item count to get to: 200000 curr count: 109095
2021/12/10 14:03:08 Waiting for src bucket item count to get to: 200000 curr count: 125778
2021/12/10 14:03:13 Waiting for src bucket item count to get to: 200000 curr count: 125778
2021/12/10 14:03:18 Waiting for src bucket item count to get to: 200000 curr count: 142423
2021/12/10 14:03:23 Waiting for src bucket item count to get to: 200000 curr count: 158899
2021/12/10 14:03:28 Waiting for src bucket item count to get to: 200000 curr count: 158899
2021/12/10 14:03:33 Waiting for src bucket item count to get to: 200000 curr count: 175901
2021/12/10 14:03:38 Waiting for src bucket item count to get to: 200000 curr count: 175901
2021/12/10 14:03:43 Waiting for src bucket item count to get to: 200000 curr count: 192533
2021/12/10 14:03:48 Waiting for src bucket item count to get to: 200000 curr count: 192533
2021/12/10 14:03:53 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 14:03:53 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 14:03:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:03:58 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 14:04:03 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 14:04:08 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 14:04:13 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 14:04:13 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/10 14:04:13 No eventing-con process running
2021/12/10 14:04:13 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/10 14:04:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:04:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:04:25 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 14:04:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:04:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:04:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:04:37 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 14:04:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (227.68s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/10 14:04:42 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:04:42 Removing node(s): [] from the cluster
2021/12/10 14:04:47 Rebalance progress: 100
2021/12/10 14:04:52 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 14:04:52 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/10 14:04:52 No eventing-con process running
2021/12/10 14:04:52 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/10 14:04:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:04:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:04:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:05:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:05:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:05:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:05:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:05:10 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:05:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 14:05:17 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 14:05:22 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 14:05:27 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 14:05:32 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 14:05:32 App: TestDuplicateWithEventingRebOut got deployed
2021/12/10 14:05:32 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:05:32 Starting bucket ops to source bucket
2021/12/10 14:05:37 Rebalance progress: 25
2021/12/10 14:05:42 Rebalance progress: 25
2021/12/10 14:05:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:05:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:05:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:05:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 14:05:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:05:47 Waiting for src bucket item count to get to: 200000 curr count: 12519
2021/12/10 14:05:52 Waiting for src bucket item count to get to: 200000 curr count: 29323
2021/12/10 14:05:57 Waiting for src bucket item count to get to: 200000 curr count: 29323
2021/12/10 14:06:02 Waiting for src bucket item count to get to: 200000 curr count: 45961
2021/12/10 14:06:07 Waiting for src bucket item count to get to: 200000 curr count: 45961
2021/12/10 14:06:12 Waiting for src bucket item count to get to: 200000 curr count: 62574
2021/12/10 14:06:17 Waiting for src bucket item count to get to: 200000 curr count: 62574
2021/12/10 14:06:22 Waiting for src bucket item count to get to: 200000 curr count: 62574
2021/12/10 14:06:27 Waiting for src bucket item count to get to: 200000 curr count: 78781
2021/12/10 14:06:32 Waiting for src bucket item count to get to: 200000 curr count: 78781
2021/12/10 14:06:37 Waiting for src bucket item count to get to: 200000 curr count: 95601
2021/12/10 14:06:42 Waiting for src bucket item count to get to: 200000 curr count: 95601
2021/12/10 14:06:47 Waiting for src bucket item count to get to: 200000 curr count: 112458
2021/12/10 14:06:52 Waiting for src bucket item count to get to: 200000 curr count: 112458
2021/12/10 14:06:57 Waiting for src bucket item count to get to: 200000 curr count: 129473
2021/12/10 14:07:02 Waiting for src bucket item count to get to: 200000 curr count: 129473
2021/12/10 14:07:07 Waiting for src bucket item count to get to: 200000 curr count: 146118
2021/12/10 14:07:12 Waiting for src bucket item count to get to: 200000 curr count: 146118
2021/12/10 14:07:17 Waiting for src bucket item count to get to: 200000 curr count: 162614
2021/12/10 14:07:22 Waiting for src bucket item count to get to: 200000 curr count: 162614
2021/12/10 14:07:27 Waiting for src bucket item count to get to: 200000 curr count: 178657
2021/12/10 14:07:32 Waiting for src bucket item count to get to: 200000 curr count: 178657
2021/12/10 14:07:37 Waiting for src bucket item count to get to: 200000 curr count: 194983
2021/12/10 14:07:42 Waiting for src bucket item count to get to: 200000 curr count: 194983
2021/12/10 14:07:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 14:07:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 14:07:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:07:52 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 14:07:57 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 14:08:02 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 14:08:07 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 14:08:07 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/10 14:08:07 No eventing-con process running
2021/12/10 14:08:07 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/10 14:08:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:08:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:08:19 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 14:08:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:08:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:08:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:08:30 Waiting for dst bucket item count to get to: 0 curr count: 82151
2021/12/10 14:08:35 Waiting for dst bucket item count to get to: 0 curr count: 82151
2021/12/10 14:08:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (238.50s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/10 14:08:45 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 14:08:45 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/10 14:08:45 No eventing-con process running
2021/12/10 14:08:45 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/10 14:08:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:08:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:08:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:08:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:08:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:08:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:09:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:09:04 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:09:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 14:09:10 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 14:09:15 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 14:09:20 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 14:09:25 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 14:09:25 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/10 14:09:25 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:09:25 Starting bucket ops to source bucket
2021/12/10 14:09:25 Removing node(s): [] from the cluster
2021/12/10 14:09:30 Rebalance progress: 25
2021/12/10 14:09:35 Rebalance progress: 25
2021/12/10 14:09:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:09:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:09:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:09:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 14:09:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:09:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:09:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:09:52 Rebalance progress: 16.66666666666667
2021/12/10 14:09:57 Rebalance progress: 16.66666666666667
2021/12/10 14:10:02 Rebalance progress: 16.66666666666667
2021/12/10 14:10:07 Rebalance progress: 57.87545787545788
2021/12/10 14:10:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:10:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:10:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:10:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:10:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 14:10:17 Rebalance progress: 16.66666666666667
2021/12/10 14:10:22 Rebalance progress: 16.66666666666667
2021/12/10 14:10:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:10:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:10:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:10:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:10:27 Waiting for src bucket item count to get to: 200000 curr count: 83165
2021/12/10 14:10:32 Waiting for src bucket item count to get to: 200000 curr count: 99800
2021/12/10 14:10:37 Waiting for src bucket item count to get to: 200000 curr count: 99800
2021/12/10 14:10:42 Waiting for src bucket item count to get to: 200000 curr count: 116579
2021/12/10 14:10:47 Waiting for src bucket item count to get to: 200000 curr count: 116579
2021/12/10 14:10:52 Waiting for src bucket item count to get to: 200000 curr count: 133095
2021/12/10 14:10:57 Waiting for src bucket item count to get to: 200000 curr count: 133095
2021/12/10 14:11:02 Waiting for src bucket item count to get to: 200000 curr count: 149593
2021/12/10 14:11:07 Waiting for src bucket item count to get to: 200000 curr count: 149593
2021/12/10 14:11:12 Waiting for src bucket item count to get to: 200000 curr count: 166415
2021/12/10 14:11:17 Waiting for src bucket item count to get to: 200000 curr count: 166415
2021/12/10 14:11:22 Waiting for src bucket item count to get to: 200000 curr count: 182744
2021/12/10 14:11:27 Waiting for src bucket item count to get to: 200000 curr count: 182744
2021/12/10 14:11:32 Waiting for src bucket item count to get to: 200000 curr count: 199690
2021/12/10 14:11:37 Waiting for src bucket item count to get to: 200000 curr count: 199690
2021/12/10 14:11:43 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 14:11:43 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 14:11:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:11:48 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 14:11:53 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 14:11:58 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 14:12:03 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 14:12:03 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/10 14:12:03 No eventing-con process running
2021/12/10 14:12:03 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/10 14:12:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:12:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:12:14 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 14:12:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:12:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:12:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:12:26 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 14:12:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (230.74s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/10 14:12:36 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 14:12:36 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/10 14:12:36 No eventing-con process running
2021/12/10 14:12:36 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/10 14:12:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:12:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:12:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:12:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:12:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:12:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:12:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:12:55 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:12:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 14:13:01 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 14:13:06 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 14:13:11 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 14:13:16 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 14:13:16 App: TestDuplicateWithKVRebIn got deployed
2021/12/10 14:13:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 14:13:16 Starting bucket ops to source bucket
2021/12/10 14:13:16 Removing node(s): [] from the cluster
2021/12/10 14:13:21 Rebalance progress: 16.66666666666667
2021/12/10 14:13:26 Rebalance progress: 16.66666666666667
2021/12/10 14:13:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:13:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:13:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:13:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:13:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:13:36 Rebalance progress: 16.66666666666667
2021/12/10 14:13:41 Rebalance progress: 16.66666666666667
2021/12/10 14:13:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:13:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:13:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:13:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:13:46 Waiting for src bucket item count to get to: 200000 curr count: 37007
2021/12/10 14:13:51 Waiting for src bucket item count to get to: 200000 curr count: 53057
2021/12/10 14:13:56 Waiting for src bucket item count to get to: 200000 curr count: 53057
2021/12/10 14:14:01 Waiting for src bucket item count to get to: 200000 curr count: 69419
2021/12/10 14:14:06 Waiting for src bucket item count to get to: 200000 curr count: 69419
2021/12/10 14:14:11 Waiting for src bucket item count to get to: 200000 curr count: 84819
2021/12/10 14:14:16 Waiting for src bucket item count to get to: 200000 curr count: 84819
2021/12/10 14:14:21 Waiting for src bucket item count to get to: 200000 curr count: 84819
2021/12/10 14:14:26 Waiting for src bucket item count to get to: 200000 curr count: 101583
2021/12/10 14:14:31 Waiting for src bucket item count to get to: 200000 curr count: 101583
2021/12/10 14:14:36 Waiting for src bucket item count to get to: 200000 curr count: 118270
2021/12/10 14:14:41 Waiting for src bucket item count to get to: 200000 curr count: 118270
2021/12/10 14:14:46 Waiting for src bucket item count to get to: 200000 curr count: 134978
2021/12/10 14:14:51 Waiting for src bucket item count to get to: 200000 curr count: 134978
2021/12/10 14:14:56 Waiting for src bucket item count to get to: 200000 curr count: 151707
2021/12/10 14:15:01 Waiting for src bucket item count to get to: 200000 curr count: 151707
2021/12/10 14:15:06 Waiting for src bucket item count to get to: 200000 curr count: 168158
2021/12/10 14:15:11 Waiting for src bucket item count to get to: 200000 curr count: 168158
2021/12/10 14:15:16 Waiting for src bucket item count to get to: 200000 curr count: 183571
2021/12/10 14:15:21 Waiting for src bucket item count to get to: 200000 curr count: 183571
2021/12/10 14:15:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 14:15:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 14:15:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:15:31 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 14:15:36 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 14:15:41 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 14:15:46 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 14:15:46 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/10 14:15:46 No eventing-con process running
2021/12/10 14:15:46 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/10 14:15:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:15:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:15:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 14:15:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:16:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:16:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:16:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (218.78s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/10 14:16:10 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 14:16:10 Removing node(s): [] from the cluster
2021/12/10 14:16:15 Rebalance progress: 100
2021/12/10 14:16:20 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 14:16:20 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/10 14:16:20 No eventing-con process running
2021/12/10 14:16:20 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/10 14:16:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:16:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:16:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:16:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:16:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:16:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:16:39 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:16:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 14:16:45 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 14:16:50 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 14:16:55 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 14:17:00 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 14:17:00 App: TestDuplicateWithKVRebOut got deployed
2021/12/10 14:17:00 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:17:00 Starting bucket ops to source bucket
2021/12/10 14:17:05 Rebalance progress: 16.66666666666667
2021/12/10 14:17:10 Rebalance progress: 16.66666666666667
2021/12/10 14:17:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:17:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:17:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:17:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:17:15 Waiting for src bucket item count to get to: 200000 curr count: 14555
2021/12/10 14:17:20 Waiting for src bucket item count to get to: 200000 curr count: 30753
2021/12/10 14:17:25 Waiting for src bucket item count to get to: 200000 curr count: 30753
2021/12/10 14:17:30 Waiting for src bucket item count to get to: 200000 curr count: 47456
2021/12/10 14:17:35 Waiting for src bucket item count to get to: 200000 curr count: 47456
2021/12/10 14:17:40 Waiting for src bucket item count to get to: 200000 curr count: 64228
2021/12/10 14:17:45 Waiting for src bucket item count to get to: 200000 curr count: 64228
2021/12/10 14:17:50 Waiting for src bucket item count to get to: 200000 curr count: 64228
2021/12/10 14:17:55 Waiting for src bucket item count to get to: 200000 curr count: 80242
2021/12/10 14:18:00 Waiting for src bucket item count to get to: 200000 curr count: 80242
2021/12/10 14:18:05 Waiting for src bucket item count to get to: 200000 curr count: 96724
2021/12/10 14:18:10 Waiting for src bucket item count to get to: 200000 curr count: 96724
2021/12/10 14:18:15 Waiting for src bucket item count to get to: 200000 curr count: 113209
2021/12/10 14:18:20 Waiting for src bucket item count to get to: 200000 curr count: 113209
2021/12/10 14:18:25 Waiting for src bucket item count to get to: 200000 curr count: 129724
2021/12/10 14:18:30 Waiting for src bucket item count to get to: 200000 curr count: 129724
2021/12/10 14:18:35 Waiting for src bucket item count to get to: 200000 curr count: 146325
2021/12/10 14:18:40 Waiting for src bucket item count to get to: 200000 curr count: 146325
2021/12/10 14:18:45 Waiting for src bucket item count to get to: 200000 curr count: 163195
2021/12/10 14:18:50 Waiting for src bucket item count to get to: 200000 curr count: 163195
2021/12/10 14:18:55 Waiting for src bucket item count to get to: 200000 curr count: 179179
2021/12/10 14:19:00 Waiting for src bucket item count to get to: 200000 curr count: 179179
2021/12/10 14:19:05 Waiting for src bucket item count to get to: 200000 curr count: 195668
2021/12/10 14:19:10 Waiting for src bucket item count to get to: 200000 curr count: 195668
2021/12/10 14:19:15 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 14:19:15 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 14:19:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:19:20 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 14:19:25 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 14:19:30 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 14:19:35 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 14:19:35 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/10 14:19:35 No eventing-con process running
2021/12/10 14:19:35 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/10 14:19:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:19:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:19:47 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 14:19:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:19:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:19:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:19:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 14:19:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (233.92s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/10 14:20:09 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 14:20:09 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/10 14:20:09 No eventing-con process running
2021/12/10 14:20:09 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/10 14:20:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:20:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:20:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:20:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:20:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:20:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:20:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:20:27 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:20:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 14:20:33 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 14:20:38 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 14:20:43 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 14:20:48 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 14:20:48 App: TestDuplicateWithKVSwapReb got deployed
2021/12/10 14:20:48 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 14:20:48 Starting bucket ops to source bucket
2021/12/10 14:20:49 Removing node(s): [] from the cluster
2021/12/10 14:20:54 Rebalance progress: 16.66666666666667
2021/12/10 14:20:59 Rebalance progress: 16.66666666666667
2021/12/10 14:21:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:21:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:21:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:21:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:21:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 14:21:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:21:09 Rebalance progress: 16.66666666666667
2021/12/10 14:21:14 Rebalance progress: 16.66666666666667
2021/12/10 14:21:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:21:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:21:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:21:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:21:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 14:21:24 Rebalance progress: 16.66666666666667
2021/12/10 14:21:29 Rebalance progress: 16.66666666666667
2021/12/10 14:21:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:21:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:21:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:21:34 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:21:34 Waiting for src bucket item count to get to: 200000 curr count: 64790
2021/12/10 14:21:39 Waiting for src bucket item count to get to: 200000 curr count: 64790
2021/12/10 14:21:44 Waiting for src bucket item count to get to: 200000 curr count: 80170
2021/12/10 14:21:49 Waiting for src bucket item count to get to: 200000 curr count: 80170
2021/12/10 14:21:54 Waiting for src bucket item count to get to: 200000 curr count: 96763
2021/12/10 14:21:59 Waiting for src bucket item count to get to: 200000 curr count: 96763
2021/12/10 14:22:04 Waiting for src bucket item count to get to: 200000 curr count: 113427
2021/12/10 14:22:09 Waiting for src bucket item count to get to: 200000 curr count: 113427
2021/12/10 14:22:14 Waiting for src bucket item count to get to: 200000 curr count: 129879
2021/12/10 14:22:19 Waiting for src bucket item count to get to: 200000 curr count: 129879
2021/12/10 14:22:24 Waiting for src bucket item count to get to: 200000 curr count: 129879
2021/12/10 14:22:29 Waiting for src bucket item count to get to: 200000 curr count: 146067
2021/12/10 14:22:34 Waiting for src bucket item count to get to: 200000 curr count: 146067
2021/12/10 14:22:39 Waiting for src bucket item count to get to: 200000 curr count: 162737
2021/12/10 14:22:44 Waiting for src bucket item count to get to: 200000 curr count: 178521
2021/12/10 14:22:49 Waiting for src bucket item count to get to: 200000 curr count: 178521
2021/12/10 14:22:54 Waiting for src bucket item count to get to: 200000 curr count: 194793
2021/12/10 14:22:59 Waiting for src bucket item count to get to: 200000 curr count: 194793
2021/12/10 14:23:04 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 14:23:04 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 14:23:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:23:09 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 14:23:14 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 14:23:20 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 14:23:25 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 14:23:25 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/10 14:23:25 No eventing-con process running
2021/12/10 14:23:25 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/10 14:23:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:23:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:23:36 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 14:23:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:23:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:23:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:23:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 14:23:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.26s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/10 14:23:58 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:23:58 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 14:23:58 No eventing-con process running
2021/12/10 14:23:58 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/10 14:24:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:24:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:24:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:24:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:24:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:24:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:24:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:24:16 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:24:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/10 14:24:23 Starting bucket ops to source bucket
2021/12/10 14:24:28 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 14:24:33 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 14:24:38 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 14:24:38 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:24:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:24:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:24:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:24:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:24:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:24:43 Rebalance progress: 16.66666666666667
2021/12/10 14:24:48 Rebalance progress: 16.66666666666667
2021/12/10 14:24:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:24:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:24:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:24:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:24:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:24:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:24:59 Rebalance progress: 16.66666666666667
2021/12/10 14:25:04 Rebalance progress: 16.66666666666667
2021/12/10 14:25:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:25:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:25:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:25:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 14:25:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 14:25:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:25:19 Rebalance progress: 12.5
2021/12/10 14:25:24 Rebalance progress: 12.5
2021/12/10 14:25:29 Rebalance progress: 12.5
2021/12/10 14:25:34 Rebalance progress: 28.40909090909091
2021/12/10 14:25:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:25:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:25:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:25:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:25:44 Rebalance progress: 12.5
2021/12/10 14:25:49 Rebalance progress: 12.5
2021/12/10 14:25:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:25:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:25:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:25:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 14:25:59 Rebalance progress: 12.5
2021/12/10 14:26:04 Rebalance progress: 12.5
2021/12/10 14:26:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:26:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:26:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:26:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 14:26:14 Rebalance progress: 12.5
2021/12/10 14:26:19 Rebalance progress: 12.5
2021/12/10 14:26:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:26:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:26:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:26:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:26:30 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:26:35 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:26:40 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:26:45 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:26:45 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 14:26:45 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:26:45 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/10 14:26:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:26:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:26:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12178
2021/12/10 14:26:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:26:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:27:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:27:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12178
2021/12/10 14:27:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.16s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/10 14:27:18 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 14:27:18 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/10 14:27:18 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:27:18 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/10 14:27:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:27:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:27:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:27:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:27:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:27:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:27:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:27:37 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:27:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/10 14:27:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 14:27:48 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/10 14:27:53 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/10 14:27:58 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/10 14:27:58 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:27:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:27:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:27:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:27:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:27:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 14:27:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:28:04 Rebalance progress: 12.5
2021/12/10 14:28:09 Rebalance progress: 12.5
2021/12/10 14:28:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:28:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:28:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:28:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 14:28:19 Rebalance progress: 12.5
2021/12/10 14:28:24 Rebalance progress: 12.5
2021/12/10 14:28:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:28:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:28:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:28:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:28:34 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 14:28:39 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 14:28:44 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 14:28:49 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 14:28:49 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/10 14:28:49 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:28:49 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/10 14:28:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:28:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:29:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4226
2021/12/10 14:29:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4226
2021/12/10 14:29:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:29:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:29:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:29:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4595
2021/12/10 14:29:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.43s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/10 14:29:28 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 14:29:28 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 14:29:28 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:29:28 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/10 14:29:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:29:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:29:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:29:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:29:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:29:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:29:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:29:46 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:29:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 14:29:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 14:29:57 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/10 14:30:02 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/10 14:30:07 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/10 14:30:07 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:30:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:30:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:30:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:30:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:30:13 Rebalance progress: 12.5
2021/12/10 14:30:18 Rebalance progress: 12.5
2021/12/10 14:30:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:30:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:30:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:30:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:30:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:30:28 Rebalance progress: 12.5
2021/12/10 14:30:33 Rebalance progress: 12.5
2021/12/10 14:30:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:30:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:30:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:30:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 14:30:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:30:44 Rebalance progress: 12.5
2021/12/10 14:30:49 Rebalance progress: 12.5
2021/12/10 14:30:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:30:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:30:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:30:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:30:59 Rebalance progress: 12.5
2021/12/10 14:31:04 Rebalance progress: 12.5
2021/12/10 14:31:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:31:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:31:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:31:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 14:31:14 Rebalance progress: 12.5
2021/12/10 14:31:19 Rebalance progress: 12.5
2021/12/10 14:31:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:31:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:31:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:31:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 14:31:29 Rebalance progress: 12.5
2021/12/10 14:31:34 Rebalance progress: 12.5
2021/12/10 14:31:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:31:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:31:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:31:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:31:44 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 14:31:49 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 14:31:54 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 14:31:59 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 14:31:59 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 14:31:59 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:31:59 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/10 14:32:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:32:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:32:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10679
2021/12/10 14:32:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10679
2021/12/10 14:32:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:32:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:32:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:32:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10677
2021/12/10 14:32:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.21s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/10 14:32:38 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 14:32:38 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/10 14:32:38 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:32:38 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/10 14:32:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:32:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:32:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:32:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:32:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:32:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:32:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:32:51 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:32:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 14:32:57 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 14:33:02 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/10 14:33:07 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/10 14:33:12 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/10 14:33:12 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:33:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:33:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:33:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:33:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:33:18 Rebalance progress: 12.5
2021/12/10 14:33:23 Rebalance progress: 12.5
2021/12/10 14:33:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:33:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:33:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:33:34 Rebalance progress: 12.5
2021/12/10 14:33:39 Rebalance progress: 12.5
2021/12/10 14:33:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:33:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:33:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:33:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 14:33:49 Rebalance progress: 12.5
2021/12/10 14:33:54 Rebalance progress: 12.5
2021/12/10 14:33:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:33:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:33:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:33:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:34:04 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 14:34:09 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 14:34:14 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 14:34:19 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 14:34:19 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/10 14:34:19 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:34:19 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/10 14:34:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:34:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:34:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6155
2021/12/10 14:34:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6155
2021/12/10 14:34:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:34:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:34:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:34:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6155
2021/12/10 14:34:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.91s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/10 14:34:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:34:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:34:58 Rebalance progress: 100
2021/12/10 14:35:03 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 14:35:03 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/10 14:35:03 No eventing-con process running
2021/12/10 14:35:03 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/10 14:35:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:35:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:35:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:35:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:35:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:35:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:35:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:35:16 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:35:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/10 14:35:23 Starting bucket ops to source bucket
2021/12/10 14:35:28 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/10 14:35:33 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/10 14:35:38 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/10 14:35:38 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:35:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:35:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:35:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:35:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:35:44 Rebalance progress: 12.5
2021/12/10 14:35:49 Rebalance progress: 12.5
2021/12/10 14:35:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:35:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:35:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:35:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/10 14:35:54 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/10 14:35:59 Rebalance progress: 12.5
2021/12/10 14:36:04 Rebalance progress: 12.5
2021/12/10 14:36:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:36:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:36:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:36:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:36:14 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 14:36:19 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 14:36:24 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 14:36:29 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 14:36:29 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/10 14:36:29 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:36:29 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/10 14:36:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:36:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:36:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3133
2021/12/10 14:36:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3133
2021/12/10 14:36:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:36:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:36:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:36:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4598
2021/12/10 14:36:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.61s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/10 14:37:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:37:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:37:08 Rebalance progress: 100
2021/12/10 14:37:13 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 14:37:13 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/10 14:37:13 No eventing-con process running
2021/12/10 14:37:13 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/10 14:37:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:37:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:37:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:37:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:37:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:37:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:37:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:37:26 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:37:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/10 14:37:32 Starting bucket ops to source bucket
2021/12/10 14:37:37 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/10 14:37:42 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/10 14:37:47 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/10 14:37:47 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:37:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:37:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:37:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:37:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:37:53 Rebalance progress: 12.5
2021/12/10 14:37:58 Rebalance progress: 12.5
2021/12/10 14:38:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:38:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:38:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:38:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/10 14:38:03 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/10 14:38:08 Rebalance progress: 12.5
2021/12/10 14:38:13 Rebalance progress: 12.5
2021/12/10 14:38:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:38:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:38:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:38:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:38:18 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/10 14:38:23 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 14:38:28 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 14:38:33 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 14:38:38 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 14:38:38 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/10 14:38:38 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:38:38 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/10 14:38:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:38:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:38:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:38:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:38:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:38:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 38080
2021/12/10 14:38:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (119.07s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/10 14:39:06 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 14:39:06 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/10 14:39:06 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:39:06 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/10 14:39:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:39:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:39:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:39:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:39:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:39:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:39:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:39:25 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:39:26 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:39:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:39:26 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:39:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 14:39:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 14:39:37 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/10 14:39:42 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/10 14:39:47 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/10 14:39:47 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/10 14:39:52 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/10 14:39:57 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/10 14:40:02 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/10 14:40:02 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:40:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:40:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:40:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:40:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:40:08 Rebalance progress: 12.5
2021/12/10 14:40:13 Rebalance progress: 12.5
2021/12/10 14:40:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:40:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:40:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:40:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:40:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:40:24 Rebalance progress: 12.5
2021/12/10 14:40:29 Rebalance progress: 12.5
2021/12/10 14:40:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:40:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:40:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:40:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 14:40:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:40:39 Rebalance progress: 12.5
2021/12/10 14:40:44 Rebalance progress: 12.5
2021/12/10 14:40:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:40:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:40:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:40:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:40:54 Rebalance progress: 12.5
2021/12/10 14:40:59 Rebalance progress: 12.5
2021/12/10 14:41:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:41:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:41:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:41:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 14:41:09 Rebalance progress: 12.5
2021/12/10 14:41:14 Rebalance progress: 12.5
2021/12/10 14:41:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:41:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:41:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:41:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 14:41:24 Rebalance progress: 12.5
2021/12/10 14:41:29 Rebalance progress: 12.5
2021/12/10 14:41:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:41:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:41:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:41:34 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:41:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 14:41:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 14:41:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 14:41:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 14:41:55 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/10 14:41:55 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:41:55 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/10 14:41:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:42:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:42:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12207
2021/12/10 14:42:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:42:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:42:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:42:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:42:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:42:20 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 14:42:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 14:42:30 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 14:42:35 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 14:42:35 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/10 14:42:35 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:42:35 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/10 14:42:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:42:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:42:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:42:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:42:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:42:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (231.93s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/10 14:42:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 14:42:58 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/10 14:42:58 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:42:58 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/10 14:43:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:43:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:43:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:43:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:43:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:43:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:43:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:43:17 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:43:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:43:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:43:18 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:43:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 14:43:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 14:43:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/10 14:43:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/10 14:43:39 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/10 14:43:39 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/10 14:43:44 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/10 14:43:49 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/10 14:43:54 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/10 14:43:54 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:43:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:43:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:43:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:43:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:43:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 14:43:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:44:01 Rebalance progress: 12.5
2021/12/10 14:44:06 Rebalance progress: 12.5
2021/12/10 14:44:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:44:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:44:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:44:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 14:44:16 Rebalance progress: 12.5
2021/12/10 14:44:21 Rebalance progress: 12.5
2021/12/10 14:44:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:44:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:44:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:44:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:44:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 14:44:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 14:44:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 14:44:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 14:44:46 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/10 14:44:46 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:44:46 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/10 14:44:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:44:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:44:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6140
2021/12/10 14:44:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:45:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:45:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:45:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5770
2021/12/10 14:45:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5770
2021/12/10 14:45:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:45:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:45:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 14:45:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 14:45:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 14:45:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 14:45:36 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/10 14:45:36 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:45:36 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/10 14:45:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:45:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:45:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:45:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:45:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:45:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.31s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/10 14:46:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 14:46:00 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/10 14:46:00 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:46:00 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/10 14:46:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:46:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:46:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:46:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:46:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:46:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:46:18 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 14:46:18 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/10 14:46:18 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:46:18 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/10 14:46:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:46:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:46:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:46:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:46:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:46:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:46:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:46:32 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:46:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:46:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:46:33 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:46:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/10 14:46:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 14:46:44 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/10 14:46:49 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/10 14:46:54 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/10 14:46:54 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/10 14:46:59 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/10 14:47:04 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/10 14:47:09 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/10 14:47:09 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:47:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:47:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:47:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:47:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:47:14 Rebalance progress: 12.5
2021/12/10 14:47:19 Rebalance progress: 12.5
2021/12/10 14:47:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:47:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:47:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:47:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:47:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:47:30 Rebalance progress: 12.5
2021/12/10 14:47:35 Rebalance progress: 12.5
2021/12/10 14:47:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:47:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:47:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:47:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 14:47:45 Rebalance progress: 12.5
2021/12/10 14:47:50 Rebalance progress: 12.5
2021/12/10 14:47:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:47:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:47:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:47:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:48:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 14:48:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 14:48:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 14:48:15 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 14:48:15 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/10 14:48:15 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:48:15 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/10 14:48:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:48:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:48:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7596
2021/12/10 14:48:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:48:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:48:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:48:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7596
2021/12/10 14:48:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7596
2021/12/10 14:48:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:48:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:48:51 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 14:48:56 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 14:49:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 14:49:06 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 14:49:06 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/10 14:49:06 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:49:06 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/10 14:49:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:49:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:49:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:49:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:49:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:49:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.22s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/10 14:49:30 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:49:30 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 14:49:30 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:49:30 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/10 14:49:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:49:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:49:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:49:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:49:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:49:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:49:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:49:48 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:49:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/10 14:49:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 14:50:00 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 14:50:05 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 14:50:10 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 14:50:10 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:50:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:50:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:50:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:50:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:50:30 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:50:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:50:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:51:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:51:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:51:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:51:06 Rebalance progress: 12.5
2021/12/10 14:51:11 Rebalance progress: 12.5
2021/12/10 14:51:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:51:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:51:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:51:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:51:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:51:21 Rebalance progress: 12.5
2021/12/10 14:51:26 Rebalance progress: 12.5
2021/12/10 14:51:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:51:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 14:51:36 Rebalance progress: 12.5
2021/12/10 14:51:41 Rebalance progress: 12.5
2021/12/10 14:51:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:51:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:51:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:51:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:51:51 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:51:57 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:52:02 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:52:07 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:52:07 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 14:52:07 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:52:07 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/10 14:52:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:52:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:52:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11175
2021/12/10 14:52:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:52:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:52:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:52:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2120
2021/12/10 14:52:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2120
2021/12/10 14:52:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.21s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/10 14:52:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:52:45 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 14:52:45 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:52:45 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/10 14:52:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:52:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:52:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:52:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:52:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:52:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:53:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:53:04 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:53:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/10 14:53:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 14:53:15 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 14:53:20 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 14:53:25 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 14:53:25 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:53:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:53:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:53:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:53:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:53:45 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:53:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:53:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:53:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:53:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:53:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:53:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:54:16 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:54:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:54:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:54:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:54:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:54:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:54:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:54:47 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:54:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:54:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:54:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:54:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:54:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:54:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:55:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:55:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:55:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:55:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:55:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:55:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:55:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:55:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:55:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:55:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:55:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:55:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:55:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:56:04 Rebalance progress: 12.5
2021/12/10 14:56:09 Rebalance progress: 12.5
2021/12/10 14:56:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:56:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:56:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:56:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:56:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:56:20 Rebalance progress: 12.5
2021/12/10 14:56:25 Rebalance progress: 12.5
2021/12/10 14:56:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:56:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:56:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:56:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 14:56:35 Rebalance progress: 12.5
2021/12/10 14:56:40 Rebalance progress: 12.5
2021/12/10 14:56:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:56:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:56:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:56:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:56:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:56:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:57:00 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:57:05 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 14:57:05 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 14:57:05 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:57:05 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/10 14:57:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:57:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:57:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21478
2021/12/10 14:57:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:57:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:57:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:57:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21478
2021/12/10 14:57:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (293.36s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/10 14:57:38 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 14:57:38 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/10 14:57:38 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:57:38 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/10 14:57:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:57:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:57:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 14:57:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:57:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:57:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:57:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 14:57:57 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 14:57:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 14:58:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 14:58:08 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/10 14:58:13 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/10 14:58:18 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/10 14:58:18 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:58:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:58:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:58:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 14:58:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:58:24 Rebalance progress: 12.5
2021/12/10 14:58:29 Rebalance progress: 12.5
2021/12/10 14:58:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:58:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:58:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:58:34 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:58:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:58:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:58:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 14:58:49 Rebalance progress: 12.5
2021/12/10 14:58:54 Rebalance progress: 12.5
2021/12/10 14:58:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:58:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:58:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:58:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 14:58:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 14:59:04 Rebalance progress: 12.5
2021/12/10 14:59:09 Rebalance progress: 12.5
2021/12/10 14:59:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:59:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:59:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:59:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 14:59:19 Rebalance progress: 12.5
2021/12/10 14:59:24 Rebalance progress: 12.5
2021/12/10 14:59:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:59:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:59:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 14:59:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 14:59:34 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 14:59:39 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 14:59:44 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 14:59:49 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 14:59:49 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/10 14:59:49 eventingSetup.go:46: No eventing-con process running
2021/12/10 14:59:49 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/10 14:59:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 14:59:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:00:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/10 15:00:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/10 15:00:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:00:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:00:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:00:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/10 15:00:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.44s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/10 15:00:28 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 15:00:28 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/10 15:00:28 eventingSetup.go:46: No eventing-con process running
2021/12/10 15:00:28 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/10 15:00:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:00:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:00:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:00:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:00:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:00:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:00:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:00:46 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 15:00:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 15:00:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 15:00:58 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/10 15:01:03 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/10 15:01:08 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/10 15:01:08 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:01:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:01:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:01:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/10 15:01:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 15:01:13 Rebalance progress: 12.5
2021/12/10 15:01:18 Rebalance progress: 12.5
2021/12/10 15:01:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:01:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:01:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:01:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/10 15:01:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 15:01:28 Rebalance progress: 12.5
2021/12/10 15:01:33 Rebalance progress: 12.5
2021/12/10 15:01:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:01:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:01:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:01:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 15:01:44 Rebalance progress: 12.5
2021/12/10 15:01:49 Rebalance progress: 12.5
2021/12/10 15:01:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:01:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:01:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:01:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 15:01:59 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 15:02:04 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 15:02:09 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 15:02:14 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 15:02:14 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/10 15:02:14 eventingSetup.go:46: No eventing-con process running
2021/12/10 15:02:14 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/10 15:02:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:02:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:02:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6095
2021/12/10 15:02:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:02:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:02:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:02:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6095
2021/12/10 15:02:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.32s)
=== RUN   TestEventingRebInWithOps
2021/12/10 15:02:47 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 15:02:47 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/10 15:02:47 eventingSetup.go:46: No eventing-con process running
2021/12/10 15:02:47 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/10 15:02:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:02:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:02:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:02:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:03:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:03:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:03:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:03:06 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 15:03:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 15:03:12 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 15:03:17 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 15:03:22 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 15:03:27 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 15:03:27 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/10 15:03:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 15:03:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 15:03:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 15:03:33 Rebalance progress: 12.5
2021/12/10 15:03:38 Rebalance progress: 12.5
2021/12/10 15:03:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:03:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:03:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:03:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 15:03:48 Rebalance progress: 12.5
2021/12/10 15:03:53 Rebalance progress: 12.5
2021/12/10 15:03:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:03:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:03:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:03:58 Waiting for src bucket item count to get to: 200000 curr count: 34177
2021/12/10 15:04:03 Waiting for src bucket item count to get to: 200000 curr count: 49987
2021/12/10 15:04:08 Waiting for src bucket item count to get to: 200000 curr count: 49987
2021/12/10 15:04:13 Waiting for src bucket item count to get to: 200000 curr count: 65932
2021/12/10 15:04:18 Waiting for src bucket item count to get to: 200000 curr count: 65932
2021/12/10 15:04:23 Waiting for src bucket item count to get to: 200000 curr count: 80780
2021/12/10 15:04:28 Waiting for src bucket item count to get to: 200000 curr count: 80780
2021/12/10 15:04:33 Waiting for src bucket item count to get to: 200000 curr count: 96869
2021/12/10 15:04:38 Waiting for src bucket item count to get to: 200000 curr count: 96869
2021/12/10 15:04:43 Waiting for src bucket item count to get to: 200000 curr count: 113139
2021/12/10 15:04:48 Waiting for src bucket item count to get to: 200000 curr count: 113139
2021/12/10 15:04:53 Waiting for src bucket item count to get to: 200000 curr count: 129389
2021/12/10 15:04:58 Waiting for src bucket item count to get to: 200000 curr count: 129389
2021/12/10 15:05:03 Waiting for src bucket item count to get to: 200000 curr count: 145509
2021/12/10 15:05:08 Waiting for src bucket item count to get to: 200000 curr count: 145509
2021/12/10 15:05:13 Waiting for src bucket item count to get to: 200000 curr count: 145509
2021/12/10 15:05:18 Waiting for src bucket item count to get to: 200000 curr count: 161660
2021/12/10 15:05:23 Waiting for src bucket item count to get to: 200000 curr count: 161660
2021/12/10 15:05:28 Waiting for src bucket item count to get to: 200000 curr count: 176352
2021/12/10 15:05:33 Waiting for src bucket item count to get to: 200000 curr count: 176352
2021/12/10 15:05:38 Waiting for src bucket item count to get to: 200000 curr count: 192669
2021/12/10 15:05:43 Waiting for src bucket item count to get to: 200000 curr count: 192669
2021/12/10 15:05:48 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 15:05:48 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 15:05:48 Starting bucket ops to source bucket
2021/12/10 15:09:33 Waiting for dst bucket item count to get to: 0 curr count: 16065
2021/12/10 15:09:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:09:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 15:09:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 15:09:38 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 15:09:43 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 15:09:48 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 15:09:53 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 15:09:53 App: TestEventingRebInWithOps got un-deployed
2021/12/10 15:09:53 No eventing-con process running
2021/12/10 15:09:53 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/10 15:09:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:10:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:10:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:10:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:10:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:10:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (449.49s)
=== RUN   TestEventingRebOutWithOps
2021/12/10 15:10:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 15:10:12 Removing node(s): [] from the cluster
2021/12/10 15:10:17 Rebalance progress: 100
2021/12/10 15:10:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 15:10:22 App: TestEventingRebOutWithOps got un-deployed
2021/12/10 15:10:22 No eventing-con process running
2021/12/10 15:10:22 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/10 15:10:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:10:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:10:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:10:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:10:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:10:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:10:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:10:41 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 15:10:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 15:10:47 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 15:10:52 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 15:10:57 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 15:11:02 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 15:11:02 App: TestEventingRebOutWithOps got deployed
2021/12/10 15:11:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 15:11:02 Starting bucket ops to source bucket
2021/12/10 15:11:07 Rebalance progress: 12.5
2021/12/10 15:11:12 Rebalance progress: 12.5
2021/12/10 15:11:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:11:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:11:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:11:17 Waiting for src bucket item count to get to: 200000 curr count: 11226
2021/12/10 15:11:22 Waiting for src bucket item count to get to: 200000 curr count: 27261
2021/12/10 15:11:27 Waiting for src bucket item count to get to: 200000 curr count: 27261
2021/12/10 15:11:32 Waiting for src bucket item count to get to: 200000 curr count: 43808
2021/12/10 15:11:37 Waiting for src bucket item count to get to: 200000 curr count: 43808
2021/12/10 15:11:42 Waiting for src bucket item count to get to: 200000 curr count: 60261
2021/12/10 15:11:47 Waiting for src bucket item count to get to: 200000 curr count: 60261
2021/12/10 15:11:52 Waiting for src bucket item count to get to: 200000 curr count: 60261
2021/12/10 15:11:57 Waiting for src bucket item count to get to: 200000 curr count: 75442
2021/12/10 15:12:02 Waiting for src bucket item count to get to: 200000 curr count: 75442
2021/12/10 15:12:07 Waiting for src bucket item count to get to: 200000 curr count: 91523
2021/12/10 15:12:12 Waiting for src bucket item count to get to: 200000 curr count: 91523
2021/12/10 15:12:17 Waiting for src bucket item count to get to: 200000 curr count: 108027
2021/12/10 15:12:22 Waiting for src bucket item count to get to: 200000 curr count: 108027
2021/12/10 15:12:27 Waiting for src bucket item count to get to: 200000 curr count: 124508
2021/12/10 15:12:32 Waiting for src bucket item count to get to: 200000 curr count: 124508
2021/12/10 15:12:37 Waiting for src bucket item count to get to: 200000 curr count: 141209
2021/12/10 15:12:42 Waiting for src bucket item count to get to: 200000 curr count: 141209
2021/12/10 15:12:47 Waiting for src bucket item count to get to: 200000 curr count: 157826
2021/12/10 15:12:52 Waiting for src bucket item count to get to: 200000 curr count: 157826
2021/12/10 15:12:57 Waiting for src bucket item count to get to: 200000 curr count: 173391
2021/12/10 15:13:02 Waiting for src bucket item count to get to: 200000 curr count: 173391
2021/12/10 15:13:07 Waiting for src bucket item count to get to: 200000 curr count: 189559
2021/12/10 15:13:12 Waiting for src bucket item count to get to: 200000 curr count: 189559
2021/12/10 15:13:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 15:13:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 15:13:17 Starting bucket ops to source bucket
2021/12/10 15:17:02 Waiting for dst bucket item count to get to: 0 curr count: 14242
2021/12/10 15:17:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:17:02 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 15:17:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 15:17:07 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 15:17:12 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 15:17:17 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 15:17:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 15:17:22 App: TestEventingRebOutWithOps got un-deployed
2021/12/10 15:17:22 No eventing-con process running
2021/12/10 15:17:22 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/10 15:17:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:17:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:17:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:17:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:17:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:17:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (449.07s)
=== RUN   TestEventingSwapRebWithOps
2021/12/10 15:17:46 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 15:17:46 App: TestEventingSwapRebWithOps got un-deployed
2021/12/10 15:17:46 No eventing-con process running
2021/12/10 15:17:46 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/10 15:17:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:17:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:17:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:17:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:17:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:17:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:18:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:18:04 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 15:18:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 15:18:10 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 15:18:15 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 15:18:20 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 15:18:25 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 15:18:25 App: TestEventingSwapRebWithOps got deployed
2021/12/10 15:18:25 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 15:18:25 Starting bucket ops to source bucket
2021/12/10 15:18:26 Removing node(s): [] from the cluster
2021/12/10 15:18:31 Rebalance progress: 12.5
2021/12/10 15:18:36 Rebalance progress: 12.5
2021/12/10 15:18:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:18:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:18:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:18:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 15:18:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 15:18:47 Rebalance progress: 12.5
2021/12/10 15:18:52 Rebalance progress: 12.5
2021/12/10 15:18:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:18:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:18:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:18:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 15:19:02 Rebalance progress: 12.5
2021/12/10 15:19:07 Rebalance progress: 12.5
2021/12/10 15:19:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:19:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:19:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:19:12 Waiting for src bucket item count to get to: 200000 curr count: 68519
2021/12/10 15:19:17 Waiting for src bucket item count to get to: 200000 curr count: 68519
2021/12/10 15:19:22 Waiting for src bucket item count to get to: 200000 curr count: 83503
2021/12/10 15:19:27 Waiting for src bucket item count to get to: 200000 curr count: 83503
2021/12/10 15:19:32 Waiting for src bucket item count to get to: 200000 curr count: 100059
2021/12/10 15:19:37 Waiting for src bucket item count to get to: 200000 curr count: 100059
2021/12/10 15:19:42 Waiting for src bucket item count to get to: 200000 curr count: 116695
2021/12/10 15:19:47 Waiting for src bucket item count to get to: 200000 curr count: 116695
2021/12/10 15:19:52 Waiting for src bucket item count to get to: 200000 curr count: 133300
2021/12/10 15:19:57 Waiting for src bucket item count to get to: 200000 curr count: 133300
2021/12/10 15:20:02 Waiting for src bucket item count to get to: 200000 curr count: 149858
2021/12/10 15:20:07 Waiting for src bucket item count to get to: 200000 curr count: 149858
2021/12/10 15:20:12 Waiting for src bucket item count to get to: 200000 curr count: 166231
2021/12/10 15:20:17 Waiting for src bucket item count to get to: 200000 curr count: 166231
2021/12/10 15:20:22 Waiting for src bucket item count to get to: 200000 curr count: 181186
2021/12/10 15:20:27 Waiting for src bucket item count to get to: 200000 curr count: 181186
2021/12/10 15:20:32 Waiting for src bucket item count to get to: 200000 curr count: 197713
2021/12/10 15:20:37 Waiting for src bucket item count to get to: 200000 curr count: 197713
2021/12/10 15:20:42 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 15:20:42 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 15:20:42 Starting bucket ops to source bucket
2021/12/10 15:24:27 Waiting for dst bucket item count to get to: 0 curr count: 5183
2021/12/10 15:24:32 Waiting for dst bucket item count to get to: 0 curr count: 5183
2021/12/10 15:24:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:24:32 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 15:24:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 15:24:37 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 15:24:42 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 15:24:47 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 15:24:52 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 15:24:52 App: TestEventingSwapRebWithOps got un-deployed
2021/12/10 15:24:52 No eventing-con process running
2021/12/10 15:24:52 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/10 15:24:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:24:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:24:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:25:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:25:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:25:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (450.09s)
=== RUN   TestKVRebInWithOps
2021/12/10 15:25:16 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 15:25:16 App: TestKVRebInWithOps got un-deployed
2021/12/10 15:25:16 No eventing-con process running
2021/12/10 15:25:16 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/10 15:25:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:25:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:25:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:25:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:25:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:25:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:25:34 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 15:25:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 15:25:40 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 15:25:45 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 15:25:51 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 15:25:56 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 15:25:56 App: TestKVRebInWithOps got deployed
2021/12/10 15:25:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 15:25:56 Starting bucket ops to source bucket
2021/12/10 15:25:56 Removing node(s): [] from the cluster
2021/12/10 15:26:02 Rebalance progress: 12.5
2021/12/10 15:26:07 Rebalance progress: 12.5
2021/12/10 15:26:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:26:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:26:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:26:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 15:26:17 Rebalance progress: 12.5
2021/12/10 15:26:22 Rebalance progress: 12.5
2021/12/10 15:26:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:26:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:26:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:26:27 Waiting for src bucket item count to get to: 200000 curr count: 36625
2021/12/10 15:26:32 Waiting for src bucket item count to get to: 200000 curr count: 52967
2021/12/10 15:26:37 Waiting for src bucket item count to get to: 200000 curr count: 52967
2021/12/10 15:26:42 Waiting for src bucket item count to get to: 200000 curr count: 69455
2021/12/10 15:26:47 Waiting for src bucket item count to get to: 200000 curr count: 69455
2021/12/10 15:26:52 Waiting for src bucket item count to get to: 200000 curr count: 84325
2021/12/10 15:26:57 Waiting for src bucket item count to get to: 200000 curr count: 84325
2021/12/10 15:27:02 Waiting for src bucket item count to get to: 200000 curr count: 100957
2021/12/10 15:27:07 Waiting for src bucket item count to get to: 200000 curr count: 100957
2021/12/10 15:27:12 Waiting for src bucket item count to get to: 200000 curr count: 117091
2021/12/10 15:27:17 Waiting for src bucket item count to get to: 200000 curr count: 117091
2021/12/10 15:27:22 Waiting for src bucket item count to get to: 200000 curr count: 133754
2021/12/10 15:27:27 Waiting for src bucket item count to get to: 200000 curr count: 133754
2021/12/10 15:27:32 Waiting for src bucket item count to get to: 200000 curr count: 150078
2021/12/10 15:27:37 Waiting for src bucket item count to get to: 200000 curr count: 150078
2021/12/10 15:27:42 Waiting for src bucket item count to get to: 200000 curr count: 166378
2021/12/10 15:27:47 Waiting for src bucket item count to get to: 200000 curr count: 166378
2021/12/10 15:27:52 Waiting for src bucket item count to get to: 200000 curr count: 166378
2021/12/10 15:27:57 Waiting for src bucket item count to get to: 200000 curr count: 181524
2021/12/10 15:28:02 Waiting for src bucket item count to get to: 200000 curr count: 181524
2021/12/10 15:28:07 Waiting for src bucket item count to get to: 200000 curr count: 197912
2021/12/10 15:28:12 Waiting for src bucket item count to get to: 200000 curr count: 197912
2021/12/10 15:28:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 15:28:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 15:28:17 Starting bucket ops to source bucket
2021/12/10 15:32:02 Waiting for dst bucket item count to get to: 0 curr count: 14030
2021/12/10 15:32:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:32:02 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 15:32:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 15:32:07 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 15:32:12 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 15:32:17 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 15:32:22 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 15:32:22 App: TestKVRebInWithOps got un-deployed
2021/12/10 15:32:22 No eventing-con process running
2021/12/10 15:32:22 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/10 15:32:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:32:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:32:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:32:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:32:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:32:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (449.67s)
=== RUN   TestKVRebOutWithOps
2021/12/10 15:32:41 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 15:32:41 Removing node(s): [] from the cluster
2021/12/10 15:32:46 Rebalance progress: 100
2021/12/10 15:32:51 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 15:32:51 App: TestKVRebOutWithOps got un-deployed
2021/12/10 15:32:51 No eventing-con process running
2021/12/10 15:32:51 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/10 15:32:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:32:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:32:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:32:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:33:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:33:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:33:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:33:09 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 15:33:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 15:33:15 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 15:33:20 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 15:33:25 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 15:33:30 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 15:33:30 App: TestKVRebOutWithOps got deployed
2021/12/10 15:33:30 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 15:33:30 Starting bucket ops to source bucket
2021/12/10 15:33:36 Rebalance progress: 12.5
2021/12/10 15:33:41 Rebalance progress: 12.5
2021/12/10 15:33:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:33:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:33:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:33:46 Waiting for src bucket item count to get to: 200000 curr count: 13564
2021/12/10 15:33:51 Waiting for src bucket item count to get to: 200000 curr count: 29804
2021/12/10 15:33:56 Waiting for src bucket item count to get to: 200000 curr count: 29804
2021/12/10 15:34:01 Waiting for src bucket item count to get to: 200000 curr count: 46109
2021/12/10 15:34:06 Waiting for src bucket item count to get to: 200000 curr count: 46109
2021/12/10 15:34:11 Waiting for src bucket item count to get to: 200000 curr count: 62573
2021/12/10 15:34:16 Waiting for src bucket item count to get to: 200000 curr count: 62573
2021/12/10 15:34:21 Waiting for src bucket item count to get to: 200000 curr count: 62573
2021/12/10 15:34:26 Waiting for src bucket item count to get to: 200000 curr count: 78121
2021/12/10 15:34:31 Waiting for src bucket item count to get to: 200000 curr count: 78121
2021/12/10 15:34:36 Waiting for src bucket item count to get to: 200000 curr count: 94184
2021/12/10 15:34:41 Waiting for src bucket item count to get to: 200000 curr count: 94184
2021/12/10 15:34:46 Waiting for src bucket item count to get to: 200000 curr count: 110709
2021/12/10 15:34:51 Waiting for src bucket item count to get to: 200000 curr count: 110709
2021/12/10 15:34:56 Waiting for src bucket item count to get to: 200000 curr count: 127299
2021/12/10 15:35:01 Waiting for src bucket item count to get to: 200000 curr count: 127299
2021/12/10 15:35:06 Waiting for src bucket item count to get to: 200000 curr count: 143633
2021/12/10 15:35:11 Waiting for src bucket item count to get to: 200000 curr count: 143633
2021/12/10 15:35:16 Waiting for src bucket item count to get to: 200000 curr count: 160170
2021/12/10 15:35:21 Waiting for src bucket item count to get to: 200000 curr count: 160170
2021/12/10 15:35:26 Waiting for src bucket item count to get to: 200000 curr count: 175550
2021/12/10 15:35:31 Waiting for src bucket item count to get to: 200000 curr count: 175550
2021/12/10 15:35:36 Waiting for src bucket item count to get to: 200000 curr count: 191619
2021/12/10 15:35:41 Waiting for src bucket item count to get to: 200000 curr count: 191619
2021/12/10 15:35:46 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 15:35:46 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 15:35:46 Starting bucket ops to source bucket
2021/12/10 15:39:31 Waiting for dst bucket item count to get to: 0 curr count: 11735
2021/12/10 15:39:36 Waiting for dst bucket item count to get to: 0 curr count: 11735
2021/12/10 15:39:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:39:36 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 15:39:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 15:39:41 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 15:39:46 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 15:39:51 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 15:39:56 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 15:39:56 App: TestKVRebOutWithOps got un-deployed
2021/12/10 15:39:56 No eventing-con process running
2021/12/10 15:39:56 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/10 15:39:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:40:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:40:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:40:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:40:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:40:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (453.87s)
=== RUN   TestKVSwapRebWithOps
2021/12/10 15:40:19 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 15:40:19 App: TestKVSwapRebWithOps got un-deployed
2021/12/10 15:40:19 No eventing-con process running
2021/12/10 15:40:19 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/10 15:40:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:40:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:40:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:40:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:40:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:40:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:40:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:40:38 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 15:40:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 15:40:44 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 15:40:49 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 15:40:54 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 15:40:59 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 15:40:59 App: TestKVSwapRebWithOps got deployed
2021/12/10 15:40:59 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 15:40:59 Starting bucket ops to source bucket
2021/12/10 15:40:59 Removing node(s): [] from the cluster
2021/12/10 15:41:04 Rebalance progress: 12.5
2021/12/10 15:41:09 Rebalance progress: 12.5
2021/12/10 15:41:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:41:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:41:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:41:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 15:41:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 15:41:20 Rebalance progress: 12.5
2021/12/10 15:41:25 Rebalance progress: 12.5
2021/12/10 15:41:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:41:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:41:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:41:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 15:41:35 Rebalance progress: 12.5
2021/12/10 15:41:40 Rebalance progress: 12.5
2021/12/10 15:41:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:41:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:41:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:41:45 Waiting for src bucket item count to get to: 200000 curr count: 62441
2021/12/10 15:41:50 Waiting for src bucket item count to get to: 200000 curr count: 62441
2021/12/10 15:41:55 Waiting for src bucket item count to get to: 200000 curr count: 77665
2021/12/10 15:42:00 Waiting for src bucket item count to get to: 200000 curr count: 77665
2021/12/10 15:42:05 Waiting for src bucket item count to get to: 200000 curr count: 93791
2021/12/10 15:42:10 Waiting for src bucket item count to get to: 200000 curr count: 93791
2021/12/10 15:42:15 Waiting for src bucket item count to get to: 200000 curr count: 110357
2021/12/10 15:42:20 Waiting for src bucket item count to get to: 200000 curr count: 126970
2021/12/10 15:42:25 Waiting for src bucket item count to get to: 200000 curr count: 126970
2021/12/10 15:42:30 Waiting for src bucket item count to get to: 200000 curr count: 143293
2021/12/10 15:42:35 Waiting for src bucket item count to get to: 200000 curr count: 143293
2021/12/10 15:42:40 Waiting for src bucket item count to get to: 200000 curr count: 159893
2021/12/10 15:42:45 Waiting for src bucket item count to get to: 200000 curr count: 159893
2021/12/10 15:42:50 Waiting for src bucket item count to get to: 200000 curr count: 159893
2021/12/10 15:42:55 Waiting for src bucket item count to get to: 200000 curr count: 175305
2021/12/10 15:43:00 Waiting for src bucket item count to get to: 200000 curr count: 175305
2021/12/10 15:43:05 Waiting for src bucket item count to get to: 200000 curr count: 191578
2021/12/10 15:43:10 Waiting for src bucket item count to get to: 200000 curr count: 191578
2021/12/10 15:43:15 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 15:43:15 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 15:43:15 Starting bucket ops to source bucket
2021/12/10 15:47:01 Waiting for dst bucket item count to get to: 0 curr count: 11557
2021/12/10 15:47:06 Waiting for dst bucket item count to get to: 0 curr count: 11557
2021/12/10 15:47:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:47:06 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 15:47:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 15:47:11 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 15:47:16 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 15:47:21 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 15:47:26 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 15:47:26 App: TestKVSwapRebWithOps got un-deployed
2021/12/10 15:47:26 No eventing-con process running
2021/12/10 15:47:26 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/10 15:47:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:47:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:47:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:47:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:47:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:47:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (449.91s)
=== RUN   TestEventingRebInWithTimers
2021/12/10 15:47:49 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 15:47:49 App: TestEventingRebInWithTimers got un-deployed
2021/12/10 15:47:49 No eventing-con process running
2021/12/10 15:47:49 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/10 15:47:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:47:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:47:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:47:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:48:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:48:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:48:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:48:08 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 15:48:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 15:48:14 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 15:48:19 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 15:48:24 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 15:48:29 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 15:48:29 App: TestEventingRebInWithTimers got deployed
2021/12/10 15:48:29 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 15:48:29 Starting bucket ops to source bucket
2021/12/10 15:48:29 Removing node(s): [] from the cluster
2021/12/10 15:48:35 Rebalance progress: 12.5
2021/12/10 15:48:40 Rebalance progress: 12.5
2021/12/10 15:48:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:48:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:48:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:48:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 15:48:50 Rebalance progress: 12.5
2021/12/10 15:48:55 Rebalance progress: 12.5
2021/12/10 15:49:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:49:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:49:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:49:00 Waiting for src bucket item count to get to: 50000 curr count: 25418
2021/12/10 15:49:05 Waiting for src bucket item count to get to: 50000 curr count: 35366
2021/12/10 15:49:10 Waiting for src bucket item count to get to: 50000 curr count: 35366
2021/12/10 15:49:15 Waiting for src bucket item count to get to: 50000 curr count: 47840
2021/12/10 15:49:20 Waiting for src bucket item count to get to: 50000 curr count: 47840
2021/12/10 15:49:30 Waiting for dst bucket item count to get to: 50000 curr count: 49005
2021/12/10 15:49:35 Waiting for dst bucket item count to get to: 50000 curr count: 49005
2021/12/10 15:49:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 15:49:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 15:49:35 Starting bucket ops to source bucket
2021/12/10 15:50:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 15:50:50 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 15:50:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 15:51:00 Waiting for dst bucket item count to get to: 0 curr count: 40819
2021/12/10 15:51:05 Waiting for dst bucket item count to get to: 0 curr count: 40819
2021/12/10 15:51:10 Waiting for dst bucket item count to get to: 0 curr count: 24102
2021/12/10 15:51:15 Waiting for dst bucket item count to get to: 0 curr count: 24102
2021/12/10 15:51:20 Waiting for dst bucket item count to get to: 0 curr count: 11928
2021/12/10 15:51:25 Waiting for dst bucket item count to get to: 0 curr count: 11928
2021/12/10 15:51:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:51:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 15:51:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 15:51:30 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 15:51:35 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 15:51:40 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 15:51:45 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 15:51:45 App: TestEventingRebInWithTimers got un-deployed
2021/12/10 15:51:45 No eventing-con process running
2021/12/10 15:51:45 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/10 15:51:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:51:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:51:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:51:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:51:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:51:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (259.00s)
=== RUN   TestEventingRebOutWithTimers
2021/12/10 15:52:03 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 15:52:04 Removing node(s): [] from the cluster
2021/12/10 15:52:09 Rebalance progress: 100
2021/12/10 15:52:14 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 15:52:14 App: TestEventingRebOutWithTimers got un-deployed
2021/12/10 15:52:14 No eventing-con process running
2021/12/10 15:52:14 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/10 15:52:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:52:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:52:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:52:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:52:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:52:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:52:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:52:32 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 15:52:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 15:52:38 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 15:52:43 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 15:52:48 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 15:52:53 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 15:52:53 App: TestEventingRebOutWithTimers got deployed
2021/12/10 15:52:53 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 15:52:53 Starting bucket ops to source bucket
2021/12/10 15:52:58 Rebalance progress: 12.5
2021/12/10 15:53:03 Rebalance progress: 12.5
2021/12/10 15:53:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:53:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:53:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:53:08 Waiting for src bucket item count to get to: 50000 curr count: 7838
2021/12/10 15:53:13 Waiting for src bucket item count to get to: 50000 curr count: 22474
2021/12/10 15:53:18 Waiting for src bucket item count to get to: 50000 curr count: 22474
2021/12/10 15:53:23 Waiting for src bucket item count to get to: 50000 curr count: 34333
2021/12/10 15:53:28 Waiting for src bucket item count to get to: 50000 curr count: 34333
2021/12/10 15:53:33 Waiting for src bucket item count to get to: 50000 curr count: 45097
2021/12/10 15:53:38 Waiting for src bucket item count to get to: 50000 curr count: 45097
2021/12/10 15:53:43 Waiting for src bucket item count to get to: 50000 curr count: 45097
2021/12/10 15:53:53 Waiting for dst bucket item count to get to: 50000 curr count: 36110
2021/12/10 15:53:58 Waiting for dst bucket item count to get to: 50000 curr count: 36110
2021/12/10 15:54:03 Waiting for dst bucket item count to get to: 50000 curr count: 46744
2021/12/10 15:54:09 Waiting for dst bucket item count to get to: 50000 curr count: 46744
2021/12/10 15:54:09 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 15:54:09 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 15:54:09 Starting bucket ops to source bucket
2021/12/10 15:55:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 15:55:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 15:55:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 15:55:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 15:55:38 Waiting for dst bucket item count to get to: 0 curr count: 34329
2021/12/10 15:55:43 Waiting for dst bucket item count to get to: 0 curr count: 34329
2021/12/10 15:55:48 Waiting for dst bucket item count to get to: 0 curr count: 23931
2021/12/10 15:55:53 Waiting for dst bucket item count to get to: 0 curr count: 23931
2021/12/10 15:55:58 Waiting for dst bucket item count to get to: 0 curr count: 5249
2021/12/10 15:56:03 Waiting for dst bucket item count to get to: 0 curr count: 5249
2021/12/10 15:56:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:56:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 15:56:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 15:56:08 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 15:56:13 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 15:56:18 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 15:56:23 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 15:56:23 App: TestEventingRebOutWithTimers got un-deployed
2021/12/10 15:56:23 No eventing-con process running
2021/12/10 15:56:23 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/10 15:56:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:56:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:56:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:56:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:56:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:56:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (278.32s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/10 15:56:47 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 15:56:47 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/10 15:56:47 No eventing-con process running
2021/12/10 15:56:47 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/10 15:56:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:56:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:56:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 15:56:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:57:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:57:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:57:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 15:57:05 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 15:57:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 15:57:11 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 15:57:16 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 15:57:21 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 15:57:26 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 15:57:26 App: TestEventingSwapRebWithTimers got deployed
2021/12/10 15:57:26 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 15:57:26 Starting bucket ops to source bucket
2021/12/10 15:57:27 Removing node(s): [] from the cluster
2021/12/10 15:57:32 Rebalance progress: 12.5
2021/12/10 15:57:37 Rebalance progress: 12.5
2021/12/10 15:57:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:57:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:57:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:57:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 15:57:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 15:57:47 Rebalance progress: 12.5
2021/12/10 15:57:52 Rebalance progress: 12.5
2021/12/10 15:57:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:57:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:57:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:57:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 15:58:02 Rebalance progress: 12.5
2021/12/10 15:58:07 Rebalance progress: 12.5
2021/12/10 15:58:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:58:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 15:58:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 15:58:17 Waiting for dst bucket item count to get to: 50000 curr count: 34966
2021/12/10 15:58:22 Waiting for dst bucket item count to get to: 50000 curr count: 34966
2021/12/10 15:58:27 Waiting for dst bucket item count to get to: 50000 curr count: 41936
2021/12/10 15:58:32 Waiting for dst bucket item count to get to: 50000 curr count: 41936
2021/12/10 15:58:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 15:58:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 15:58:32 Starting bucket ops to source bucket
2021/12/10 15:59:42 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 15:59:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 15:59:52 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 15:59:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:00:02 Waiting for dst bucket item count to get to: 0 curr count: 40689
2021/12/10 16:00:07 Waiting for dst bucket item count to get to: 0 curr count: 40689
2021/12/10 16:00:12 Waiting for dst bucket item count to get to: 0 curr count: 31266
2021/12/10 16:00:17 Waiting for dst bucket item count to get to: 0 curr count: 31266
2021/12/10 16:00:22 Waiting for dst bucket item count to get to: 0 curr count: 9815
2021/12/10 16:00:27 Waiting for dst bucket item count to get to: 0 curr count: 9815
2021/12/10 16:00:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:00:27 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 16:00:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:00:32 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 16:00:37 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 16:00:42 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 16:00:47 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 16:00:47 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/10 16:00:47 No eventing-con process running
2021/12/10 16:00:47 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/10 16:00:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:00:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:00:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:00:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:01:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:01:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (263.74s)
=== RUN   TestKVRebInWithTimers
2021/12/10 16:01:10 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 16:01:10 App: TestKVRebInWithTimers got un-deployed
2021/12/10 16:01:10 No eventing-con process running
2021/12/10 16:01:10 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/10 16:01:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:01:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:01:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:01:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:01:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:01:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:01:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:01:29 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:01:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 16:01:35 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 16:01:40 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 16:01:45 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 16:01:50 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 16:01:50 App: TestKVRebInWithTimers got deployed
2021/12/10 16:01:50 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:01:50 Starting bucket ops to source bucket
2021/12/10 16:01:51 Removing node(s): [] from the cluster
2021/12/10 16:01:56 Rebalance progress: 12.5
2021/12/10 16:02:01 Rebalance progress: 12.5
2021/12/10 16:02:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:02:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:02:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:02:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:02:11 Rebalance progress: 12.5
2021/12/10 16:02:16 Rebalance progress: 12.5
2021/12/10 16:02:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:02:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:02:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:02:21 Waiting for src bucket item count to get to: 50000 curr count: 35888
2021/12/10 16:02:26 Waiting for src bucket item count to get to: 50000 curr count: 35888
2021/12/10 16:02:31 Waiting for src bucket item count to get to: 50000 curr count: 47281
2021/12/10 16:02:36 Waiting for src bucket item count to get to: 50000 curr count: 47281
2021/12/10 16:02:46 Waiting for dst bucket item count to get to: 50000 curr count: 41984
2021/12/10 16:02:51 Waiting for dst bucket item count to get to: 50000 curr count: 41984
2021/12/10 16:02:51 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:02:51 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:02:51 Starting bucket ops to source bucket
2021/12/10 16:04:00 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:04:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:04:10 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:04:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:04:20 Waiting for dst bucket item count to get to: 0 curr count: 42976
2021/12/10 16:04:25 Waiting for dst bucket item count to get to: 0 curr count: 42976
2021/12/10 16:04:30 Waiting for dst bucket item count to get to: 0 curr count: 19956
2021/12/10 16:04:35 Waiting for dst bucket item count to get to: 0 curr count: 19956
2021/12/10 16:04:40 Waiting for dst bucket item count to get to: 0 curr count: 7594
2021/12/10 16:04:45 Waiting for dst bucket item count to get to: 0 curr count: 7594
2021/12/10 16:04:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:04:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 16:04:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:04:50 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 16:04:55 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 16:05:00 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 16:05:05 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 16:05:05 App: TestKVRebInWithTimers got un-deployed
2021/12/10 16:05:05 No eventing-con process running
2021/12/10 16:05:05 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/10 16:05:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:05:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:05:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:05:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:05:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:05:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (258.30s)
=== RUN   TestKVRebOutWithTimers
2021/12/10 16:05:24 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:05:24 Removing node(s): [] from the cluster
2021/12/10 16:05:29 Rebalance progress: 100
2021/12/10 16:05:34 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 16:05:34 App: TestKVRebOutWithTimers got un-deployed
2021/12/10 16:05:34 No eventing-con process running
2021/12/10 16:05:34 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/10 16:05:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:05:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:05:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:05:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:05:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:05:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:05:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:05:53 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:05:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 16:05:59 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 16:06:04 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 16:06:09 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 16:06:14 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 16:06:14 App: TestKVRebOutWithTimers got deployed
2021/12/10 16:06:14 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:06:14 Starting bucket ops to source bucket
2021/12/10 16:06:19 Rebalance progress: 12.5
2021/12/10 16:06:24 Rebalance progress: 12.5
2021/12/10 16:06:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:06:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:06:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:06:29 Waiting for src bucket item count to get to: 50000 curr count: 7501
2021/12/10 16:06:34 Waiting for src bucket item count to get to: 50000 curr count: 20398
2021/12/10 16:06:39 Waiting for src bucket item count to get to: 50000 curr count: 20398
2021/12/10 16:06:44 Waiting for src bucket item count to get to: 50000 curr count: 31623
2021/12/10 16:06:49 Waiting for src bucket item count to get to: 50000 curr count: 31623
2021/12/10 16:06:54 Waiting for src bucket item count to get to: 50000 curr count: 31623
2021/12/10 16:06:59 Waiting for src bucket item count to get to: 50000 curr count: 43073
2021/12/10 16:07:04 Waiting for src bucket item count to get to: 50000 curr count: 43073
2021/12/10 16:07:14 Waiting for dst bucket item count to get to: 50000 curr count: 41442
2021/12/10 16:07:19 Waiting for dst bucket item count to get to: 50000 curr count: 41442
2021/12/10 16:07:19 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:07:19 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:07:19 Starting bucket ops to source bucket
2021/12/10 16:08:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:08:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:08:39 Waiting for dst bucket item count to get to: 0 curr count: 46824
2021/12/10 16:08:44 Waiting for dst bucket item count to get to: 0 curr count: 46824
2021/12/10 16:08:49 Waiting for dst bucket item count to get to: 0 curr count: 39247
2021/12/10 16:08:54 Waiting for dst bucket item count to get to: 0 curr count: 39247
2021/12/10 16:08:59 Waiting for dst bucket item count to get to: 0 curr count: 14625
2021/12/10 16:09:04 Waiting for dst bucket item count to get to: 0 curr count: 14625
2021/12/10 16:09:09 Waiting for dst bucket item count to get to: 0 curr count: 36
2021/12/10 16:09:14 Waiting for dst bucket item count to get to: 0 curr count: 36
2021/12/10 16:09:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:09:14 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 16:09:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:09:19 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 16:09:24 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 16:09:29 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 16:09:34 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 16:09:34 App: TestKVRebOutWithTimers got un-deployed
2021/12/10 16:09:34 No eventing-con process running
2021/12/10 16:09:34 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/10 16:09:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:09:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:09:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:09:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:09:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:09:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (268.61s)
=== RUN   TestKVSwapRebWithTimers
2021/12/10 16:09:57 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 16:09:57 App: TestKVSwapRebWithTimers got un-deployed
2021/12/10 16:09:57 No eventing-con process running
2021/12/10 16:09:57 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/10 16:09:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:10:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:10:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:10:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:10:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:10:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:10:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:10:16 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:10:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 16:10:22 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 16:10:27 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 16:10:32 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 16:10:37 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 16:10:37 App: TestKVSwapRebWithTimers got deployed
2021/12/10 16:10:37 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:10:37 Starting bucket ops to source bucket
2021/12/10 16:10:37 Removing node(s): [] from the cluster
2021/12/10 16:10:42 Rebalance progress: 12.5
2021/12/10 16:10:47 Rebalance progress: 12.5
2021/12/10 16:10:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:10:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:10:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:10:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 16:10:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:10:58 Rebalance progress: 12.5
2021/12/10 16:11:03 Rebalance progress: 12.5
2021/12/10 16:11:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:11:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:11:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:11:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 16:11:13 Rebalance progress: 12.5
2021/12/10 16:11:18 Rebalance progress: 12.5
2021/12/10 16:11:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:11:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:11:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:11:28 Waiting for dst bucket item count to get to: 50000 curr count: 31772
2021/12/10 16:11:33 Waiting for dst bucket item count to get to: 50000 curr count: 31772
2021/12/10 16:11:38 Waiting for dst bucket item count to get to: 50000 curr count: 41900
2021/12/10 16:11:43 Waiting for dst bucket item count to get to: 50000 curr count: 41900
2021/12/10 16:11:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:11:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:11:43 Starting bucket ops to source bucket
2021/12/10 16:12:52 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:12:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:13:02 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:13:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:13:12 Waiting for dst bucket item count to get to: 0 curr count: 39992
2021/12/10 16:13:17 Waiting for dst bucket item count to get to: 0 curr count: 39992
2021/12/10 16:13:22 Waiting for dst bucket item count to get to: 0 curr count: 30596
2021/12/10 16:13:27 Waiting for dst bucket item count to get to: 0 curr count: 30596
2021/12/10 16:13:32 Waiting for dst bucket item count to get to: 0 curr count: 12339
2021/12/10 16:13:37 Waiting for dst bucket item count to get to: 0 curr count: 12339
2021/12/10 16:13:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:13:37 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 16:13:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:13:42 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 16:13:47 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 16:13:52 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 16:13:58 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 16:14:03 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 16:14:03 App: TestKVSwapRebWithTimers got un-deployed
2021/12/10 16:14:03 No eventing-con process running
2021/12/10 16:14:03 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/10 16:14:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:14:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:14:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:14:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:14:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:14:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (268.81s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/10 16:14:26 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 16:14:26 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/10 16:14:26 No eventing-con process running
2021/12/10 16:14:26 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/10 16:14:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:14:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:14:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:14:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:14:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:14:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:14:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:14:45 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:14:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 16:14:51 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 16:14:56 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 16:15:01 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 16:15:06 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 16:15:06 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/10 16:15:06 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 16:15:06 Starting bucket ops to source bucket
2021/12/10 16:15:06 Removing node(s): [] from the cluster
2021/12/10 16:15:11 Rebalance progress: 12.5
2021/12/10 16:15:16 Rebalance progress: 12.5
2021/12/10 16:15:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:15:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:15:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:15:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:15:26 Rebalance progress: 12.5
2021/12/10 16:15:31 Rebalance progress: 12.5
2021/12/10 16:15:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:15:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:15:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:15:36 Waiting for src bucket item count to get to: 50000 curr count: 30134
2021/12/10 16:15:41 Waiting for src bucket item count to get to: 50000 curr count: 41441
2021/12/10 16:15:46 Waiting for src bucket item count to get to: 50000 curr count: 41441
2021/12/10 16:15:56 Waiting for dst bucket item count to get to: 50000 curr count: 33297
2021/12/10 16:16:01 Waiting for dst bucket item count to get to: 50000 curr count: 33297
2021/12/10 16:16:06 Waiting for dst bucket item count to get to: 50000 curr count: 40244
2021/12/10 16:16:11 Waiting for dst bucket item count to get to: 50000 curr count: 40244
2021/12/10 16:16:11 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:16:11 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:16:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:16:16 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 16:16:21 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 16:16:26 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 16:16:32 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 16:16:37 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 16:16:42 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 16:16:42 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/10 16:16:42 No eventing-con process running
2021/12/10 16:16:42 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/10 16:16:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:16:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:16:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:16:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:16:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:17:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:17:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:17:10 Waiting for dst bucket item count to get to: 0 curr count: 11462
2021/12/10 16:17:15 Waiting for dst bucket item count to get to: 0 curr count: 11462
2021/12/10 16:17:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.07s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/10 16:17:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 16:17:20 Removing node(s): [] from the cluster
2021/12/10 16:17:25 Rebalance progress: 100
2021/12/10 16:17:30 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 16:17:30 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/10 16:17:30 No eventing-con process running
2021/12/10 16:17:31 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/10 16:17:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:17:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:17:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:17:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:17:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:17:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:17:49 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:17:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 16:17:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 16:18:00 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 16:18:05 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 16:18:10 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 16:18:10 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/10 16:18:10 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:18:10 Starting bucket ops to source bucket
2021/12/10 16:18:15 Rebalance progress: 12.5
2021/12/10 16:18:20 Rebalance progress: 12.5
2021/12/10 16:18:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:18:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:18:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:18:25 Waiting for src bucket item count to get to: 50000 curr count: 12021
2021/12/10 16:18:30 Waiting for src bucket item count to get to: 50000 curr count: 26134
2021/12/10 16:18:35 Waiting for src bucket item count to get to: 50000 curr count: 26134
2021/12/10 16:18:40 Waiting for src bucket item count to get to: 50000 curr count: 38227
2021/12/10 16:18:45 Waiting for src bucket item count to get to: 50000 curr count: 38227
2021/12/10 16:18:50 Waiting for src bucket item count to get to: 50000 curr count: 38227
2021/12/10 16:18:55 Waiting for src bucket item count to get to: 50000 curr count: 48372
2021/12/10 16:19:00 Waiting for src bucket item count to get to: 50000 curr count: 48372
2021/12/10 16:19:10 Waiting for dst bucket item count to get to: 50000 curr count: 34835
2021/12/10 16:19:15 Waiting for dst bucket item count to get to: 50000 curr count: 34835
2021/12/10 16:19:20 Waiting for dst bucket item count to get to: 50000 curr count: 44301
2021/12/10 16:19:25 Waiting for dst bucket item count to get to: 50000 curr count: 44301
2021/12/10 16:19:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:19:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:19:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:19:30 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 16:19:36 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 16:19:41 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 16:19:46 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 16:19:51 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 16:19:56 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 16:19:56 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/10 16:19:56 No eventing-con process running
2021/12/10 16:19:56 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/10 16:19:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:20:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:20:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:20:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:20:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:20:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:20:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:20:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (184.14s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/10 16:20:29 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 16:20:29 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/10 16:20:29 No eventing-con process running
2021/12/10 16:20:29 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/10 16:20:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:20:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:20:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:20:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:20:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:20:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:20:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:20:48 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:20:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 16:20:54 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 16:20:59 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 16:21:04 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 16:21:09 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 16:21:09 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/10 16:21:09 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 16:21:09 Starting bucket ops to source bucket
2021/12/10 16:21:10 Removing node(s): [] from the cluster
2021/12/10 16:21:15 Rebalance progress: 12.5
2021/12/10 16:21:20 Rebalance progress: 12.5
2021/12/10 16:21:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:21:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:21:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:21:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 16:21:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:21:30 Rebalance progress: 12.5
2021/12/10 16:21:35 Rebalance progress: 12.5
2021/12/10 16:21:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:21:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:21:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:21:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 16:21:45 Rebalance progress: 12.5
2021/12/10 16:21:50 Rebalance progress: 12.5
2021/12/10 16:21:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:21:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:21:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:21:55 Waiting for src bucket item count to get to: 50000 curr count: 46084
2021/12/10 16:22:00 Waiting for src bucket item count to get to: 50000 curr count: 46084
2021/12/10 16:22:10 Waiting for dst bucket item count to get to: 50000 curr count: 39258
2021/12/10 16:22:15 Waiting for dst bucket item count to get to: 50000 curr count: 39258
2021/12/10 16:22:20 Waiting for dst bucket item count to get to: 50000 curr count: 46363
2021/12/10 16:22:25 Waiting for dst bucket item count to get to: 50000 curr count: 46363
2021/12/10 16:22:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:22:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:22:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:22:31 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 16:22:36 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 16:22:41 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 16:22:46 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 16:22:51 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 16:22:56 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 16:22:56 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/10 16:22:56 No eventing-con process running
2021/12/10 16:22:56 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/10 16:22:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:23:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:23:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:23:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:23:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:23:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:23:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (175.09s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/10 16:23:24 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 16:23:24 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/10 16:23:24 No eventing-con process running
2021/12/10 16:23:24 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/10 16:23:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:23:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:23:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:23:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:23:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:23:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:23:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:23:43 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:23:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 16:23:49 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 16:23:54 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 16:23:59 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 16:24:04 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 16:24:04 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/10 16:24:04 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:24:04 Starting bucket ops to source bucket
2021/12/10 16:24:05 Removing node(s): [] from the cluster
2021/12/10 16:24:10 Rebalance progress: 12.5
2021/12/10 16:24:15 Rebalance progress: 12.5
2021/12/10 16:24:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:24:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:24:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:24:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:24:25 Rebalance progress: 12.5
2021/12/10 16:24:30 Rebalance progress: 12.5
2021/12/10 16:24:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:24:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:24:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:24:35 Waiting for src bucket item count to get to: 50000 curr count: 31977
2021/12/10 16:24:40 Waiting for src bucket item count to get to: 50000 curr count: 31977
2021/12/10 16:24:45 Waiting for src bucket item count to get to: 50000 curr count: 42421
2021/12/10 16:24:50 Waiting for src bucket item count to get to: 50000 curr count: 42421
2021/12/10 16:25:00 Waiting for dst bucket item count to get to: 50000 curr count: 33716
2021/12/10 16:25:05 Waiting for dst bucket item count to get to: 50000 curr count: 33716
2021/12/10 16:25:10 Waiting for dst bucket item count to get to: 50000 curr count: 40812
2021/12/10 16:25:15 Waiting for dst bucket item count to get to: 50000 curr count: 40812
2021/12/10 16:25:15 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:25:15 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:25:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:25:20 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 16:25:25 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 16:25:30 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 16:25:35 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 16:25:40 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 16:25:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 16:25:45 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/10 16:25:45 No eventing-con process running
2021/12/10 16:25:45 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/10 16:25:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:25:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:25:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:25:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:25:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:26:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:26:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:26:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (174.26s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/10 16:26:14 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:26:14 Removing node(s): [] from the cluster
2021/12/10 16:26:19 Rebalance progress: 100
2021/12/10 16:26:24 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 16:26:24 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/10 16:26:24 No eventing-con process running
2021/12/10 16:26:24 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/10 16:26:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:26:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:26:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:26:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:26:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:26:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:26:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:26:43 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:26:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 16:26:49 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 16:26:54 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 16:26:59 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 16:27:04 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 16:27:04 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/10 16:27:04 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:27:04 Starting bucket ops to source bucket
2021/12/10 16:27:09 Rebalance progress: 12.5
2021/12/10 16:27:14 Rebalance progress: 12.5
2021/12/10 16:27:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:27:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:27:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:27:19 Waiting for src bucket item count to get to: 50000 curr count: 6873
2021/12/10 16:27:24 Waiting for src bucket item count to get to: 50000 curr count: 20595
2021/12/10 16:27:29 Waiting for src bucket item count to get to: 50000 curr count: 20595
2021/12/10 16:27:34 Waiting for src bucket item count to get to: 50000 curr count: 34692
2021/12/10 16:27:39 Waiting for src bucket item count to get to: 50000 curr count: 34692
2021/12/10 16:27:44 Waiting for src bucket item count to get to: 50000 curr count: 34692
2021/12/10 16:27:49 Waiting for src bucket item count to get to: 50000 curr count: 45413
2021/12/10 16:27:54 Waiting for src bucket item count to get to: 50000 curr count: 45413
2021/12/10 16:28:04 Waiting for dst bucket item count to get to: 50000 curr count: 29145
2021/12/10 16:28:09 Waiting for dst bucket item count to get to: 50000 curr count: 29145
2021/12/10 16:28:14 Waiting for dst bucket item count to get to: 50000 curr count: 38506
2021/12/10 16:28:19 Waiting for dst bucket item count to get to: 50000 curr count: 38506
2021/12/10 16:28:19 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:28:19 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:28:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:28:24 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 16:28:29 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 16:28:34 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 16:28:39 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 16:28:44 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 16:28:49 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 16:28:49 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/10 16:28:49 No eventing-con process running
2021/12/10 16:28:49 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/10 16:28:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:28:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:29:01 Waiting for dst bucket item count to get to: 0 curr count: 47866
2021/12/10 16:29:06 Waiting for dst bucket item count to get to: 0 curr count: 47866
2021/12/10 16:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:29:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:29:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:29:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:29:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.08s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/10 16:29:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 16:29:28 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/10 16:29:28 No eventing-con process running
2021/12/10 16:29:28 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/10 16:29:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:29:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:29:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:29:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:29:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:29:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:29:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:29:46 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:29:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 16:29:52 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 16:29:57 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 16:30:02 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 16:30:07 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 16:30:07 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/10 16:30:07 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:30:07 Starting bucket ops to source bucket
2021/12/10 16:30:08 Removing node(s): [] from the cluster
2021/12/10 16:30:13 Rebalance progress: 12.5
2021/12/10 16:30:18 Rebalance progress: 12.5
2021/12/10 16:30:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:30:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:30:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:30:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 16:30:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:30:29 Rebalance progress: 12.5
2021/12/10 16:30:34 Rebalance progress: 12.5
2021/12/10 16:30:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:30:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:30:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:30:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 16:30:44 Rebalance progress: 12.5
2021/12/10 16:30:49 Rebalance progress: 12.5
2021/12/10 16:30:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:30:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:30:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:30:54 Waiting for src bucket item count to get to: 50000 curr count: 48258
2021/12/10 16:30:59 Waiting for src bucket item count to get to: 50000 curr count: 48258
2021/12/10 16:31:09 Waiting for dst bucket item count to get to: 50000 curr count: 40034
2021/12/10 16:31:14 Waiting for dst bucket item count to get to: 50000 curr count: 40034
2021/12/10 16:31:19 Waiting for dst bucket item count to get to: 50000 curr count: 47528
2021/12/10 16:31:24 Waiting for dst bucket item count to get to: 50000 curr count: 47528
2021/12/10 16:31:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:31:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:31:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:31:29 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 16:31:34 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 16:31:39 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 16:31:44 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 16:31:49 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 16:31:54 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 16:31:54 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/10 16:31:54 No eventing-con process running
2021/12/10 16:31:54 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/10 16:31:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:32:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:32:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:32:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:32:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:32:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:32:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:32:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.92s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/10 16:32:28 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 16:32:28 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/10 16:32:28 No eventing-con process running
2021/12/10 16:32:28 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/10 16:32:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:32:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:32:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:32:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:32:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:32:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:32:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:32:46 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:32:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 16:32:52 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 16:32:57 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 16:33:02 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 16:33:07 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 16:33:07 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/10 16:33:07 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 16:33:07 Starting bucket ops to source bucket
2021/12/10 16:33:08 Removing node(s): [] from the cluster
2021/12/10 16:33:13 Rebalance progress: 12.5
2021/12/10 16:33:18 Rebalance progress: 12.5
2021/12/10 16:33:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:33:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:33:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:33:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:33:28 Rebalance progress: 12.5
2021/12/10 16:33:33 Rebalance progress: 12.5
2021/12/10 16:33:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:33:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:33:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:33:38 Waiting for src bucket item count to get to: 50000 curr count: 28017
2021/12/10 16:33:43 Waiting for src bucket item count to get to: 50000 curr count: 38071
2021/12/10 16:33:48 Waiting for src bucket item count to get to: 50000 curr count: 38071
2021/12/10 16:33:58 Waiting for dst bucket item count to get to: 50000 curr count: 29730
2021/12/10 16:34:03 Waiting for dst bucket item count to get to: 50000 curr count: 29730
2021/12/10 16:34:08 Waiting for dst bucket item count to get to: 50000 curr count: 44239
2021/12/10 16:34:13 Waiting for dst bucket item count to get to: 50000 curr count: 44239
2021/12/10 16:34:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:34:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:34:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:34:18 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 16:34:23 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 16:34:29 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 16:34:34 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 16:34:39 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 16:34:44 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 16:34:44 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/10 16:34:44 No eventing-con process running
2021/12/10 16:34:44 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/10 16:34:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:34:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:34:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:34:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:34:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:35:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:35:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:35:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.51s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/10 16:35:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 16:35:13 Removing node(s): [] from the cluster
2021/12/10 16:35:18 Rebalance progress: 100
2021/12/10 16:35:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 16:35:23 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/10 16:35:23 No eventing-con process running
2021/12/10 16:35:23 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/10 16:35:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:35:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:35:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:35:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:35:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:35:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:35:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:35:41 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:35:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 16:35:47 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/10 16:35:52 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/10 16:35:57 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/10 16:35:57 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/10 16:35:57 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:35:57 Starting bucket ops to source bucket
2021/12/10 16:36:02 Rebalance progress: 12.5
2021/12/10 16:36:07 Rebalance progress: 12.5
2021/12/10 16:36:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:36:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:36:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:36:12 Waiting for src bucket item count to get to: 50000 curr count: 16463
2021/12/10 16:36:17 Waiting for src bucket item count to get to: 50000 curr count: 16463
2021/12/10 16:36:22 Waiting for src bucket item count to get to: 50000 curr count: 27775
2021/12/10 16:36:27 Waiting for src bucket item count to get to: 50000 curr count: 27775
2021/12/10 16:36:32 Waiting for src bucket item count to get to: 50000 curr count: 39450
2021/12/10 16:36:37 Waiting for src bucket item count to get to: 50000 curr count: 39450
2021/12/10 16:36:47 Waiting for dst bucket item count to get to: 50000 curr count: 32575
2021/12/10 16:36:52 Waiting for dst bucket item count to get to: 50000 curr count: 32575
2021/12/10 16:36:57 Waiting for dst bucket item count to get to: 50000 curr count: 32575
2021/12/10 16:37:02 Waiting for dst bucket item count to get to: 50000 curr count: 42421
2021/12/10 16:37:07 Waiting for dst bucket item count to get to: 50000 curr count: 42421
2021/12/10 16:37:07 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:37:07 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:37:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:37:13 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 16:37:18 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 16:37:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 16:37:28 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 16:37:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 16:37:38 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 16:37:38 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/10 16:37:38 No eventing-con process running
2021/12/10 16:37:38 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/10 16:37:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:37:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:37:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:37:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:37:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:37:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:37:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (169.20s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/10 16:38:06 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 16:38:06 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/10 16:38:06 No eventing-con process running
2021/12/10 16:38:06 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/10 16:38:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:38:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:38:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:38:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:38:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:38:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:38:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:38:25 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:38:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 16:38:31 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 16:38:36 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 16:38:41 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 16:38:46 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 16:38:46 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/10 16:38:46 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 16:38:46 Starting bucket ops to source bucket
2021/12/10 16:38:46 Removing node(s): [] from the cluster
2021/12/10 16:38:51 Rebalance progress: 12.5
2021/12/10 16:38:56 Rebalance progress: 12.5
2021/12/10 16:39:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:39:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:39:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:39:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 16:39:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:39:07 Rebalance progress: 12.5
2021/12/10 16:39:12 Rebalance progress: 12.5
2021/12/10 16:39:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:39:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:39:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:39:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 16:39:22 Rebalance progress: 12.5
2021/12/10 16:39:27 Rebalance progress: 12.5
2021/12/10 16:39:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:39:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:39:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:39:32 Waiting for src bucket item count to get to: 50000 curr count: 48577
2021/12/10 16:39:37 Waiting for src bucket item count to get to: 50000 curr count: 48577
2021/12/10 16:39:47 Waiting for dst bucket item count to get to: 50000 curr count: 47137
2021/12/10 16:39:52 Waiting for dst bucket item count to get to: 50000 curr count: 47137
2021/12/10 16:39:52 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:39:52 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:39:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:39:57 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 16:40:02 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 16:40:07 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 16:40:12 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 16:40:17 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 16:40:22 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 16:40:22 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/10 16:40:22 No eventing-con process running
2021/12/10 16:40:22 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/10 16:40:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:40:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:40:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:40:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:40:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:40:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:40:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:40:51 Waiting for dst bucket item count to get to: 0 curr count: 34301
2021/12/10 16:40:56 Waiting for dst bucket item count to get to: 0 curr count: 34301
2021/12/10 16:40:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (179.29s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/10 16:41:06 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 16:41:06 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/10 16:41:06 No eventing-con process running
2021/12/10 16:41:06 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/10 16:41:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:41:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:41:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:41:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:41:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:41:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:41:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:41:24 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:41:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 16:41:30 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 16:41:35 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 16:41:40 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 16:41:45 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 16:41:45 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/10 16:41:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:41:45 Starting bucket ops to source bucket
2021/12/10 16:41:46 Removing node(s): [] from the cluster
2021/12/10 16:41:51 Rebalance progress: 12.5
2021/12/10 16:41:56 Rebalance progress: 12.5
2021/12/10 16:42:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:42:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:42:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:42:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:42:06 Rebalance progress: 12.5
2021/12/10 16:42:11 Rebalance progress: 12.5
2021/12/10 16:42:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:42:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:42:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:42:16 Waiting for src bucket item count to get to: 50000 curr count: 30097
2021/12/10 16:42:21 Waiting for src bucket item count to get to: 50000 curr count: 41345
2021/12/10 16:42:26 Waiting for src bucket item count to get to: 50000 curr count: 41345
2021/12/10 16:42:36 Waiting for dst bucket item count to get to: 50000 curr count: 34741
2021/12/10 16:42:41 Waiting for dst bucket item count to get to: 50000 curr count: 34741
2021/12/10 16:42:46 Waiting for dst bucket item count to get to: 50000 curr count: 44370
2021/12/10 16:42:51 Waiting for dst bucket item count to get to: 50000 curr count: 44370
2021/12/10 16:42:51 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:42:51 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:42:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:42:56 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 16:43:01 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 16:43:06 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 16:43:11 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 16:43:16 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 16:43:21 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 16:43:21 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/10 16:43:21 No eventing-con process running
2021/12/10 16:43:21 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/10 16:43:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:43:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:43:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:43:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:43:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:43:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:43:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:43:50 Waiting for dst bucket item count to get to: 0 curr count: 563
2021/12/10 16:43:55 Waiting for dst bucket item count to get to: 0 curr count: 563
2021/12/10 16:43:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (179.30s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/10 16:44:00 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:44:00 Removing node(s): [] from the cluster
2021/12/10 16:44:05 Rebalance progress: 100
2021/12/10 16:44:10 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 16:44:10 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/10 16:44:10 No eventing-con process running
2021/12/10 16:44:10 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/10 16:44:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:44:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:44:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:44:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:44:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:44:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:44:29 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:44:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 16:44:35 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 16:44:40 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 16:44:45 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 16:44:50 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 16:44:50 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/10 16:44:50 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:44:50 Starting bucket ops to source bucket
2021/12/10 16:44:55 Rebalance progress: 12.5
2021/12/10 16:45:00 Rebalance progress: 12.5
2021/12/10 16:45:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:45:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:45:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:45:05 Waiting for src bucket item count to get to: 50000 curr count: 12437
2021/12/10 16:45:10 Waiting for src bucket item count to get to: 50000 curr count: 24430
2021/12/10 16:45:15 Waiting for src bucket item count to get to: 50000 curr count: 24430
2021/12/10 16:45:20 Waiting for src bucket item count to get to: 50000 curr count: 34256
2021/12/10 16:45:25 Waiting for src bucket item count to get to: 50000 curr count: 34256
2021/12/10 16:45:30 Waiting for src bucket item count to get to: 50000 curr count: 34256
2021/12/10 16:45:35 Waiting for src bucket item count to get to: 50000 curr count: 45820
2021/12/10 16:45:40 Waiting for src bucket item count to get to: 50000 curr count: 45820
2021/12/10 16:45:50 Waiting for dst bucket item count to get to: 50000 curr count: 45069
2021/12/10 16:45:55 Waiting for dst bucket item count to get to: 50000 curr count: 45069
2021/12/10 16:45:55 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:45:55 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:45:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:46:00 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 16:46:05 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 16:46:10 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 16:46:15 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 16:46:20 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 16:46:25 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 16:46:26 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/10 16:46:26 No eventing-con process running
2021/12/10 16:46:26 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/10 16:46:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:46:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:46:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:46:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:46:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:46:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:46:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:46:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (174.14s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/10 16:46:59 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 16:46:59 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/10 16:46:59 No eventing-con process running
2021/12/10 16:46:59 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/10 16:47:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:47:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:47:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:47:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:47:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:47:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:47:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:47:18 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:47:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 16:47:24 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 16:47:29 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 16:47:34 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 16:47:39 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 16:47:39 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/10 16:47:39 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:47:39 Starting bucket ops to source bucket
2021/12/10 16:47:39 Removing node(s): [] from the cluster
2021/12/10 16:47:44 Rebalance progress: 12.5
2021/12/10 16:47:49 Rebalance progress: 12.5
2021/12/10 16:47:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:47:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:47:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:47:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 16:47:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:48:00 Rebalance progress: 12.5
2021/12/10 16:48:05 Rebalance progress: 12.5
2021/12/10 16:48:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:48:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:48:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:48:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 16:48:15 Rebalance progress: 12.5
2021/12/10 16:48:20 Rebalance progress: 12.5
2021/12/10 16:48:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:48:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:48:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:48:25 Waiting for src bucket item count to get to: 50000 curr count: 46401
2021/12/10 16:48:30 Waiting for src bucket item count to get to: 50000 curr count: 46401
2021/12/10 16:48:40 Waiting for dst bucket item count to get to: 50000 curr count: 46688
2021/12/10 16:48:45 Waiting for dst bucket item count to get to: 50000 curr count: 46688
2021/12/10 16:48:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 16:48:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 16:48:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:48:51 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 16:48:56 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 16:49:01 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 16:49:06 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 16:49:11 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 16:49:11 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/10 16:49:11 No eventing-con process running
2021/12/10 16:49:11 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/10 16:49:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:49:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:49:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:49:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 16:49:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:49:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:49:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:49:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (165.28s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/10 16:49:44 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:49:44 App: bucket_op_on_update got un-deployed
2021/12/10 16:49:44 No eventing-con process running
2021/12/10 16:49:44 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 16:49:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:49:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:49:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:49:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:49:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:49:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:50:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:50:03 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:50:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 16:50:09 Starting bucket ops to source bucket
2021/12/10 16:50:14 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:50:19 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:50:24 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:50:24 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:50:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:50:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:50:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:50:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 16:50:29 Rebalance progress: 12.5
2021/12/10 16:50:34 Rebalance progress: 12.5
2021/12/10 16:50:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:50:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:50:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:50:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 16:50:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 16:50:45 Rebalance progress: 12.5
2021/12/10 16:50:50 Rebalance progress: 12.5
2021/12/10 16:50:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:50:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:50:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:50:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 16:50:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 16:51:00 Rebalance progress: 12.5
2021/12/10 16:51:05 Rebalance progress: 12.5
2021/12/10 16:51:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:51:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:51:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:51:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:51:15 Rebalance progress: 12.5
2021/12/10 16:51:20 Rebalance progress: 12.5
2021/12/10 16:51:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:51:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:51:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:51:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 16:51:30 Rebalance progress: 12.5
2021/12/10 16:51:35 Rebalance progress: 12.5
2021/12/10 16:51:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:51:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:51:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:51:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 16:51:46 Rebalance progress: 12.5
2021/12/10 16:51:51 Rebalance progress: 12.5
2021/12/10 16:51:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:51:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:51:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:51:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:52:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:52:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:52:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:52:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:52:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:52:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:52:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 16:52:26 eventingSetup.go:46: No eventing-con process running
2021/12/10 16:52:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 16:52:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:52:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:52:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10648
2021/12/10 16:52:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:52:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:52:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:52:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (190.09s)
=== RUN   TestKVForceFailover
2021/12/10 16:52:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:52:59 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 16:52:59 eventingSetup.go:46: No eventing-con process running
2021/12/10 16:52:59 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 16:53:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:53:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:53:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:53:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:53:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:53:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:53:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:53:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 16:53:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:53:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:53:18 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:53:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 16:53:19 Starting bucket ops to source bucket
2021/12/10 16:53:24 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:53:29 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:53:34 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:53:39 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:53:39 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:53:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:53:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:53:39 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/10 16:54:00 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/10 16:54:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 16:54:05 Rebalance progress: 12.5
2021/12/10 16:54:10 Rebalance progress: 12.5
2021/12/10 16:54:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:54:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:54:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:54:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/10 16:54:20 Rebalance progress: 12.5
2021/12/10 16:54:25 Rebalance progress: 12.5
2021/12/10 16:54:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:54:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:54:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:54:30 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:54:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:54:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:54:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:54:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:54:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:55:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:55:00 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 16:55:00 eventingSetup.go:46: No eventing-con process running
2021/12/10 16:55:00 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 16:55:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:55:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:55:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7051
2021/12/10 16:55:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7051
2021/12/10 16:55:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:55:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:55:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:55:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7051
2021/12/10 16:55:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (164.37s)
=== RUN   TestKVHardFailoverRecovery
2021/12/10 16:55:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:55:39 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 16:55:39 eventingSetup.go:46: No eventing-con process running
2021/12/10 16:55:39 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 16:55:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:55:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:55:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:55:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:55:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:55:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:55:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:55:57 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:55:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 16:56:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 16:56:08 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:56:13 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:56:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:56:19 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:56:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:56:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:56:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:56:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 16:56:24 Rebalance progress: 12.5
2021/12/10 16:56:29 Rebalance progress: 12.5
2021/12/10 16:56:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:56:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:56:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:56:34 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/10 16:57:34 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/10 16:57:39 Rebalance progress: 12.5
2021/12/10 16:57:44 Rebalance progress: 12.5
2021/12/10 16:57:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:57:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:57:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:57:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 16:57:54 Rebalance progress: 12.5
2021/12/10 16:57:59 Rebalance progress: 12.5
2021/12/10 16:58:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:58:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:58:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:58:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 16:58:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:58:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:58:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:58:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:58:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:58:29 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 16:58:29 eventingSetup.go:46: No eventing-con process running
2021/12/10 16:58:29 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 16:58:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:58:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:58:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12042
2021/12/10 16:58:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12042
2021/12/10 16:58:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:58:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:58:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:58:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12042
2021/12/10 16:58:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.26s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/10 16:59:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 16:59:08 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 16:59:08 eventingSetup.go:46: No eventing-con process running
2021/12/10 16:59:08 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 16:59:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:59:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:59:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 16:59:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:59:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:59:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:59:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 16:59:27 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 16:59:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 16:59:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 16:59:38 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:59:43 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:59:48 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 16:59:48 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:59:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 16:59:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 16:59:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 16:59:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 16:59:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 16:59:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 16:59:54 Rebalance progress: 12.5
2021/12/10 16:59:59 Rebalance progress: 12.5
2021/12/10 17:00:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:00:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:00:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:00:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 17:00:09 Rebalance progress: 12.5
2021/12/10 17:00:14 Rebalance progress: 12.5
2021/12/10 17:00:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:00:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:00:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:00:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:00:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:00:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:00:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:00:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:00:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:00:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:00:49 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:00:49 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:00:49 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 17:00:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:00:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:01:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3806
2021/12/10 17:01:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3806
2021/12/10 17:01:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:01:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:01:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:01:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4592
2021/12/10 17:01:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (139.46s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/10 17:01:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:01:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:01:28 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:01:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 17:01:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:01:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:01:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:01:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:01:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:01:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:01:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:01:41 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:01:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 17:01:47 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 17:01:52 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:01:57 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:02:02 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:02:02 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:02:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:02:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:02:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 17:02:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 17:02:07 Rebalance progress: 12.5
2021/12/10 17:02:12 Rebalance progress: 12.5
2021/12/10 17:02:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:02:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:02:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:02:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 17:02:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 17:02:23 Rebalance progress: 12.5
2021/12/10 17:02:28 Rebalance progress: 12.5
2021/12/10 17:02:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:02:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:02:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:02:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 17:02:38 Rebalance progress: 12.5
2021/12/10 17:02:43 Rebalance progress: 12.5
2021/12/10 17:02:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:02:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:02:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:02:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:02:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:02:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:03:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:03:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:03:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:03:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:03:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:03:18 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:03:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 17:03:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:03:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:03:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1801
2021/12/10 17:03:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1801
2021/12/10 17:03:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:03:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:03:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:03:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6080
2021/12/10 17:03:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:03:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:03:52 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:03:52 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:03:52 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 17:03:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:03:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:03:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:04:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:04:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:04:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.03s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/10 17:04:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:04:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:04:16 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:04:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 17:04:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:04:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:04:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:04:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:04:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:04:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:04:34 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:04:34 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 17:04:34 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:04:34 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/10 17:04:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:04:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:04:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:04:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:04:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:04:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:04:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:04:48 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:04:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:04:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:04:49 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:04:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/10 17:04:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 17:05:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:05:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:05:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:05:10 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/10 17:05:15 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 17:05:20 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 17:05:25 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 17:05:25 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:05:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:05:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:05:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 17:05:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 17:05:30 Rebalance progress: 12.5
2021/12/10 17:05:35 Rebalance progress: 12.5
2021/12/10 17:05:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:05:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:05:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:05:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 17:05:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 17:05:46 Rebalance progress: 12.5
2021/12/10 17:05:51 Rebalance progress: 12.5
2021/12/10 17:05:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:05:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:05:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:05:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 17:05:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 17:06:01 Rebalance progress: 12.5
2021/12/10 17:06:06 Rebalance progress: 12.5
2021/12/10 17:06:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:06:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:06:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:06:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 17:06:16 Rebalance progress: 12.5
2021/12/10 17:06:21 Rebalance progress: 12.5
2021/12/10 17:06:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:06:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:06:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:06:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 17:06:31 Rebalance progress: 12.5
2021/12/10 17:06:36 Rebalance progress: 12.5
2021/12/10 17:06:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:06:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:06:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:06:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 17:06:47 Rebalance progress: 12.5
2021/12/10 17:06:52 Rebalance progress: 12.5
2021/12/10 17:06:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:06:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:06:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:06:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:07:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:07:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:07:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:07:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:07:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:07:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:07:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:07:27 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:07:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 17:07:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:07:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:07:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2237
2021/12/10 17:07:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2237
2021/12/10 17:07:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:07:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:07:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:07:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12160
2021/12/10 17:07:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:07:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:08:04 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:08:09 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:08:14 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:08:19 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:08:24 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:08:29 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:08:29 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 17:08:29 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:08:29 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/10 17:08:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:08:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:08:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:08:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:08:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:08:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (277.42s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/10 17:08:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:08:53 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:08:53 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:08:53 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 17:08:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:09:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:09:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:09:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:09:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:09:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:09:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:09:11 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 17:09:12 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:09:12 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/10 17:09:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:09:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:09:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:09:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:09:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:09:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:09:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:09:25 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:09:26 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:09:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:09:26 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:09:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/10 17:09:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 17:09:37 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:09:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:09:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:09:47 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/10 17:09:52 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 17:09:57 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 17:10:02 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 17:10:02 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:10:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:10:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:10:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 17:10:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 17:10:08 Rebalance progress: 12.5
2021/12/10 17:10:13 Rebalance progress: 12.5
2021/12/10 17:10:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:10:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:10:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:10:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 17:10:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 17:10:23 Rebalance progress: 12.5
2021/12/10 17:10:28 Rebalance progress: 12.5
2021/12/10 17:10:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:10:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:10:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:10:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 17:10:38 Rebalance progress: 12.5
2021/12/10 17:10:43 Rebalance progress: 12.5
2021/12/10 17:10:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:10:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:10:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:10:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:10:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:10:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:11:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:11:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:11:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:11:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:11:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:11:18 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:11:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 17:11:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:11:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:11:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4818
2021/12/10 17:11:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4818
2021/12/10 17:11:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:11:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:11:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:11:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7565
2021/12/10 17:11:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:11:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:11:54 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:11:59 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:12:04 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:12:09 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:12:14 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:12:19 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 17:12:19 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 17:12:19 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:12:19 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/10 17:12:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:12:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:12:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:12:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:12:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:12:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (229.54s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/10 17:12:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:12:43 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:12:43 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:12:43 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 17:12:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:12:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:12:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:12:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:12:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:12:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:13:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:13:01 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:13:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 17:13:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 17:13:12 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:13:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:13:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:13:22 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:13:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:13:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:13:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 17:13:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 17:13:43 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:13:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:13:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:13:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:13:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:13:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 17:13:58 Rebalance progress: 12.5
2021/12/10 17:14:03 Rebalance progress: 12.5
2021/12/10 17:14:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:14:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:14:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:14:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 17:14:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 17:14:14 Rebalance progress: 12.5
2021/12/10 17:14:19 Rebalance progress: 12.5
2021/12/10 17:14:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:14:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:14:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:14:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 17:14:29 Rebalance progress: 12.5
2021/12/10 17:14:34 Rebalance progress: 12.5
2021/12/10 17:14:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:14:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:14:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:14:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:14:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:14:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:14:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:14:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:15:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:15:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:15:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:15:09 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:15:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 17:15:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:15:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:15:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8775
2021/12/10 17:15:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8775
2021/12/10 17:15:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:15:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:15:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:15:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9164
2021/12/10 17:15:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.12s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/10 17:15:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:15:48 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:15:48 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:15:48 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 17:15:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:15:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:15:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:15:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:16:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:16:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:16:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:16:06 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:16:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 17:16:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 17:16:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:16:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:16:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:16:27 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:16:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:16:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:16:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 17:16:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 17:16:33 Rebalance progress: 12.5
2021/12/10 17:16:38 Rebalance progress: 12.5
2021/12/10 17:16:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:16:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:16:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:16:43 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:16:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:16:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:16:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 17:16:58 Rebalance progress: 12.5
2021/12/10 17:17:03 Rebalance progress: 12.5
2021/12/10 17:17:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:17:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:17:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:17:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 17:17:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 17:17:13 Rebalance progress: 12.5
2021/12/10 17:17:18 Rebalance progress: 12.5
2021/12/10 17:17:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:17:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:17:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:17:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 17:17:28 Rebalance progress: 12.5
2021/12/10 17:17:33 Rebalance progress: 12.5
2021/12/10 17:17:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:17:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:17:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:17:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:17:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:17:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:17:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:17:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:18:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:18:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:18:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:18:09 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:18:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 17:18:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:18:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:18:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4493
2021/12/10 17:18:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4493
2021/12/10 17:18:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:18:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:18:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:18:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8594
2021/12/10 17:18:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.56s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/10 17:18:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 17:18:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 17:18:48 Rebalance progress: 100
2021/12/10 17:18:48 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:19:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:19:18 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:19:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 17:19:24 Removing node(s): [] from the cluster
2021/12/10 17:19:24 Starting bucket ops to source bucket
2021/12/10 17:19:29 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/10 17:19:34 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:19:39 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 17:19:39 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:19:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:19:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 17:19:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:19:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:19:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:19:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:19:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:20:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:20:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 17:20:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 17:20:09 eventingSetup.go:46: No eventing-con process running
2021/12/10 17:20:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 17:20:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:20:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:20:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1505
2021/12/10 17:20:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1505
2021/12/10 17:20:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:20:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 17:20:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 17:20:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1505
2021/12/10 17:20:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.73s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	11995.421s

Rebalance tests took 3 hours, 19 minutes and 57 seconds.

Testing Succeeded


Version: versions-10.12.2021-09.40.cfg
Build Log: make-10.12.2021-09.40.log
Server Log: logs-10.12.2021-09.40.tar.gz

Finished