Building

Started building at 2021/12/09 10:07:43
Calculating base
Updating mirror
Basing run on 7.1.0-1873 976832bac
Updating tree for run 09.12.2021-10.07
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at efc7250, changes since last good build: none
kv_engine is at 7eb5198, changes since last good build: none
ns_server is at 26701c8, changes since last good build: 
 26701c89f MB-49883 prevent chronicle_kv_log from leaking values
 a069932d1 MB-49619: Server groups are only supported on Enterprise Edition.
 f5e762d4e MB-49883 prevent ns_couchdb_chronicle_dup from leaking values
 5fab404cf new base CSS for DWB
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 2dbd194, changes since last good build: none
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at de9f5f4, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 990bec55] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 5c2c6bad] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable bb297a8b] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable fdd8e281] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 0883bb02] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable 9b24c7ed] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable 544e0aaf] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable 5ba9c675] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable e061001] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/09 10:24:09
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/09 10:25:41 Started functional
2021/12/09 10:25:49 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/09 10:25:55 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/09 10:25:55 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 10:26:00 Removing node(s): [] from the cluster
2021/12/09 10:26:05 Rebalance progress: 0
2021/12/09 10:26:10 Rebalance progress: 1.139322916666667
2021/12/09 10:26:15 Rebalance progress: 5.696614583333333
2021/12/09 10:26:21 Rebalance progress: 9.25925925925926
2021/12/09 10:26:25 Rebalance progress: 13.88888888888889
2021/12/09 10:26:30 Rebalance progress: 18.51851851851852
2021/12/09 10:26:35 Rebalance progress: 21.99074074074074
2021/12/09 10:26:40 Rebalance progress: 26.62037037037037
2021/12/09 10:26:45 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/09 10:26:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:27:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:27:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:27:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:27:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:28:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:28:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:28:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:28:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:29:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:29:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:29:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:29:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 10:30:06 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:30:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/09 10:30:12 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 10:30:17 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 10:30:22 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 10:30:27 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 10:30:27 App: TestAdvancedGetOps got deployed
2021/12/09 10:30:27 Starting bucket ops to source bucket
2021/12/09 10:30:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 10:30:38 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/09 10:30:43 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/09 10:30:43 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 10:30:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:30:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:30:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:30:48 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 10:30:53 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 10:30:58 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 10:31:03 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 10:31:03 App: TestAdvancedGetOps got un-deployed
2021/12/09 10:31:03 No eventing-con process running
2021/12/09 10:31:03 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/09 10:31:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:31:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:31:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:31:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:31:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:31:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (275.12s)
=== RUN   TestAdvancedInsertOps
2021/12/09 10:31:26 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:31:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 10:31:32 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 10:31:37 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 10:31:42 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 10:31:47 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 10:31:47 App: TestAdvancedInsertOps got deployed
2021/12/09 10:31:47 Starting bucket ops to source bucket
2021/12/09 10:31:53 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 10:31:58 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/09 10:32:03 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/09 10:32:03 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/09 10:32:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:32:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:32:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:32:08 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 10:32:13 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 10:32:18 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 10:32:23 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 10:32:23 App: TestAdvancedInsertOps got un-deployed
2021/12/09 10:32:23 No eventing-con process running
2021/12/09 10:32:24 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/09 10:32:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:32:35 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 10:32:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:32:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:32:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 10:32:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:32:47 Testing insert operation on source bucket
2021/12/09 10:32:47 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:32:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 10:32:53 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 10:32:58 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 10:33:03 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 10:33:08 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 10:33:08 App: TestAdvancedInsertOps got deployed
2021/12/09 10:33:08 Starting bucket ops to source bucket
2021/12/09 10:33:14 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/09 10:33:19 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/09 10:33:19 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/09 10:33:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:33:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:33:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:33:24 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 10:33:29 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 10:33:34 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 10:33:39 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 10:33:39 App: TestAdvancedInsertOps got un-deployed
2021/12/09 10:33:39 No eventing-con process running
2021/12/09 10:33:39 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/09 10:33:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:33:51 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/09 10:33:56 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/09 10:33:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:33:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:34:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.39s)
=== RUN   TestAdvancedUpsertOps
2021/12/09 10:34:03 Starting bucket ops to source bucket
2021/12/09 10:34:03 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:34:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 10:34:09 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 10:34:14 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 10:34:19 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 10:34:24 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 10:34:24 App: TestAdvancedUpsertOps got deployed
2021/12/09 10:34:24 Starting bucket ops to source bucket
2021/12/09 10:34:31 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 10:34:36 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/09 10:34:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:34:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:34:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:34:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:34:41 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 10:34:46 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 10:34:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 10:34:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 10:34:56 App: TestAdvancedUpsertOps got un-deployed
2021/12/09 10:34:56 No eventing-con process running
2021/12/09 10:34:56 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/09 10:34:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:35:07 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/09 10:35:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:35:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:35:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:35:14 Testing upsert operation on source bucket
2021/12/09 10:35:14 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:35:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 10:35:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 10:35:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 10:35:30 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 10:35:35 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 10:35:35 App: TestAdvancedUpsertOps got deployed
2021/12/09 10:35:35 Starting bucket ops to source bucket
2021/12/09 10:35:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:35:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:35:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:35:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:35:41 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 10:35:46 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 10:35:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 10:35:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 10:35:56 App: TestAdvancedUpsertOps got un-deployed
2021/12/09 10:35:56 No eventing-con process running
2021/12/09 10:35:56 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/09 10:35:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:36:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:36:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:36:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (127.74s)
=== RUN   TestAdvancedReplaceOps
2021/12/09 10:36:10 Starting bucket ops to source bucket
2021/12/09 10:36:11 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:36:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/09 10:36:17 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 10:36:22 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 10:36:27 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 10:36:32 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 10:36:32 App: TestAdvancedReplaceOps got deployed
2021/12/09 10:36:32 Starting bucket ops to source bucket
2021/12/09 10:36:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 10:36:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 10:36:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:36:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:36:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:36:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:36:49 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 10:36:54 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 10:36:59 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 10:37:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 10:37:04 App: TestAdvancedReplaceOps got un-deployed
2021/12/09 10:37:04 No eventing-con process running
2021/12/09 10:37:04 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/09 10:37:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:37:15 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 10:37:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:37:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:37:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:37:22 Testing upsert operation on source bucket
2021/12/09 10:37:22 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:37:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/09 10:37:28 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 10:37:33 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 10:37:38 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 10:37:43 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 10:37:43 App: TestAdvancedReplaceOps got deployed
2021/12/09 10:37:43 Starting bucket ops to source bucket
2021/12/09 10:37:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:37:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:37:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:37:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:37:50 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 10:37:55 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 10:38:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 10:38:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 10:38:05 App: TestAdvancedReplaceOps got un-deployed
2021/12/09 10:38:05 No eventing-con process running
2021/12/09 10:38:05 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/09 10:38:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:38:16 Waiting for dst bucket item count to get to: 0 curr count: 357
2021/12/09 10:38:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:38:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:38:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (132.62s)
=== RUN   TestAdvancedDeleteOps
2021/12/09 10:38:23 Starting bucket ops to source bucket
2021/12/09 10:38:23 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:38:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/09 10:38:29 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 10:38:34 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 10:38:39 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 10:38:44 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 10:38:44 App: TestAdvancedDeleteOps got deployed
2021/12/09 10:38:44 Starting bucket ops to source bucket
2021/12/09 10:38:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:38:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 10:38:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:38:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:38:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:38:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:39:00 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 10:39:05 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 10:39:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 10:39:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 10:39:15 App: TestAdvancedDeleteOps got un-deployed
2021/12/09 10:39:15 No eventing-con process running
2021/12/09 10:39:15 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/09 10:39:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:39:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:39:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:39:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:39:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:39:33 Testing delete operation on source bucket
2021/12/09 10:39:33 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:39:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/09 10:39:39 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 10:39:44 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 10:39:49 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 10:39:54 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 10:39:54 App: TestAdvancedDeleteOps got deployed
2021/12/09 10:39:54 Starting bucket ops to source bucket
2021/12/09 10:39:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:39:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:39:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:39:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:40:00 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 10:40:05 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 10:40:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 10:40:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 10:40:15 App: TestAdvancedDeleteOps got un-deployed
2021/12/09 10:40:15 No eventing-con process running
2021/12/09 10:40:15 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/09 10:40:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:40:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:40:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:40:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.64s)
=== RUN   TestEnoentAdvancedGet
2021/12/09 10:40:29 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:40:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/09 10:40:35 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 10:40:40 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 10:40:45 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 10:40:50 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 10:40:50 App: TestEnoentAdvancedGet got deployed
2021/12/09 10:40:50 Starting bucket ops to source bucket
2021/12/09 10:40:55 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 10:41:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 10:41:05 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/09 10:41:05 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 10:41:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:41:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:41:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:41:10 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 10:41:15 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 10:41:20 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 10:41:25 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 10:41:25 App: TestEnoentAdvancedGet got un-deployed
2021/12/09 10:41:25 No eventing-con process running
2021/12/09 10:41:25 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/09 10:41:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:41:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:41:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:41:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:41:49 Waiting for dst bucket item count to get to: 0 curr count: 93
2021/12/09 10:41:54 Waiting for dst bucket item count to get to: 0 curr count: 43
2021/12/09 10:41:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:41:54 Testing get enoent operation on source bucket
2021/12/09 10:41:54 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:41:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/09 10:42:00 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 10:42:05 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 10:42:10 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 10:42:15 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 10:42:15 App: TestEnoentAdvancedGet got deployed
2021/12/09 10:42:15 Starting bucket ops to source bucket
2021/12/09 10:42:20 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/09 10:42:25 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/09 10:42:25 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/09 10:42:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:42:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:42:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:42:30 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 10:42:35 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 10:42:40 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 10:42:45 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 10:42:45 App: TestEnoentAdvancedGet got un-deployed
2021/12/09 10:42:45 No eventing-con process running
2021/12/09 10:42:45 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/09 10:42:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:42:57 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 10:42:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:42:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:43:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.95s)
=== RUN   TestEnoentAdvancedReplace
2021/12/09 10:43:04 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:43:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 10:43:10 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 10:43:15 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 10:43:20 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 10:43:25 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 10:43:25 App: TestEnoentAdvancedReplace got deployed
2021/12/09 10:43:25 Starting bucket ops to source bucket
2021/12/09 10:43:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 10:43:35 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/09 10:43:35 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 10:43:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:43:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:43:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:43:40 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 10:43:45 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 10:43:50 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 10:43:55 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 10:43:55 App: TestEnoentAdvancedReplace got un-deployed
2021/12/09 10:43:55 No eventing-con process running
2021/12/09 10:43:55 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/09 10:43:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:44:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:44:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:44:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:44:18 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/09 10:44:23 Waiting for dst bucket item count to get to: 0 curr count: 33
2021/12/09 10:44:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:44:23 Testing get enoent operation on source bucket
2021/12/09 10:44:23 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:44:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 10:44:30 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 10:44:35 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 10:44:40 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 10:44:45 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 10:44:45 App: TestEnoentAdvancedReplace got deployed
2021/12/09 10:44:45 Starting bucket ops to source bucket
2021/12/09 10:44:50 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/09 10:44:55 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/09 10:44:55 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/09 10:44:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:44:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:44:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:45:00 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 10:45:05 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 10:45:10 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 10:45:15 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 10:45:15 App: TestEnoentAdvancedReplace got un-deployed
2021/12/09 10:45:15 No eventing-con process running
2021/12/09 10:45:15 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/09 10:45:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:45:27 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 10:45:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:45:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:45:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (149.83s)
=== RUN   TestEnoentAdvancedDelete
2021/12/09 10:45:33 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:45:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 10:45:40 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 10:45:45 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 10:45:50 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 10:45:55 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 10:45:55 App: TestEnoentAdvancedDelete got deployed
2021/12/09 10:45:55 Starting bucket ops to source bucket
2021/12/09 10:46:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 10:46:05 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/09 10:46:05 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 10:46:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:46:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:46:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:46:10 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 10:46:15 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 10:46:20 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 10:46:25 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 10:46:25 App: TestEnoentAdvancedDelete got un-deployed
2021/12/09 10:46:25 No eventing-con process running
2021/12/09 10:46:25 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/09 10:46:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:46:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:46:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:46:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:46:48 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/09 10:46:53 Waiting for dst bucket item count to get to: 0 curr count: 33
2021/12/09 10:46:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:46:53 Testing get enoent operation on source bucket
2021/12/09 10:46:53 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:46:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 10:47:00 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 10:47:05 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 10:47:10 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 10:47:15 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 10:47:15 App: TestEnoentAdvancedDelete got deployed
2021/12/09 10:47:15 Starting bucket ops to source bucket
2021/12/09 10:47:20 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/09 10:47:25 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/09 10:47:25 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/09 10:47:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:47:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:47:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:47:30 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 10:47:35 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 10:47:40 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 10:47:45 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 10:47:45 App: TestEnoentAdvancedDelete got un-deployed
2021/12/09 10:47:45 No eventing-con process running
2021/12/09 10:47:45 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/09 10:47:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:47:57 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 10:47:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:47:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:48:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.88s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/09 10:48:03 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:48:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/09 10:48:10 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 10:48:15 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 10:48:20 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 10:48:25 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 10:48:25 App: TestAdvancedInsertKeyExist got deployed
2021/12/09 10:48:25 Starting bucket ops to source bucket
2021/12/09 10:48:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:48:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:48:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:48:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:48:30 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 10:48:35 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 10:48:40 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 10:48:45 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 10:48:45 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/09 10:48:45 No eventing-con process running
2021/12/09 10:48:45 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/09 10:48:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:48:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:48:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:48:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:48:59 Pumping document in the destination bucket
2021/12/09 10:48:59 Starting bucket ops to source bucket
2021/12/09 10:48:59 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:49:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/09 10:49:05 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 10:49:10 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 10:49:15 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 10:49:20 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 10:49:20 App: TestAdvancedInsertKeyExist got deployed
2021/12/09 10:49:20 Starting bucket ops to source bucket
2021/12/09 10:49:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:49:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:49:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:49:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:49:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:49:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:49:36 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 10:49:41 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 10:49:46 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 10:49:51 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 10:49:51 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/09 10:49:51 No eventing-con process running
2021/12/09 10:49:51 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/09 10:49:52 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:50:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:50:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 10:50:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:50:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:50:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.71s)
=== RUN   TestExipryGet
2021/12/09 10:50:14 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:50:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/09 10:50:20 Waiting for app: TestExipryGet to get deployed
2021/12/09 10:50:25 Waiting for app: TestExipryGet to get deployed
2021/12/09 10:50:30 Waiting for app: TestExipryGet to get deployed
2021/12/09 10:50:35 Waiting for app: TestExipryGet to get deployed
2021/12/09 10:50:35 App: TestExipryGet got deployed
2021/12/09 10:50:35 Starting bucket ops to source bucket
2021/12/09 10:51:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:51:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:51:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:51:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:51:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:51:26 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 10:51:31 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 10:51:36 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 10:51:41 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 10:51:41 App: TestExipryGet got un-deployed
2021/12/09 10:51:41 No eventing-con process running
2021/12/09 10:51:41 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/09 10:51:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:51:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:51:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:51:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:51:55 Pumping document in the destination bucket for expiry get
2021/12/09 10:51:55 Starting bucket ops to source bucket
2021/12/09 10:51:55 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:51:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/09 10:52:01 Waiting for app: TestExipryGet to get deployed
2021/12/09 10:52:06 Waiting for app: TestExipryGet to get deployed
2021/12/09 10:52:11 Waiting for app: TestExipryGet to get deployed
2021/12/09 10:52:16 Waiting for app: TestExipryGet to get deployed
2021/12/09 10:52:16 App: TestExipryGet got deployed
2021/12/09 10:52:16 Starting bucket ops to source bucket
2021/12/09 10:53:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:53:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:53:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:53:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:53:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:53:06 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 10:53:11 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 10:53:16 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 10:53:21 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 10:53:21 App: TestExipryGet got un-deployed
2021/12/09 10:53:21 No eventing-con process running
2021/12/09 10:53:21 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/09 10:53:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:53:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:53:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 10:53:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:53:40 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:53:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (210.78s)
=== RUN   TestExipryInsert
2021/12/09 10:53:45 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:53:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/09 10:53:51 Waiting for app: TestExipryInsert to get deployed
2021/12/09 10:53:56 Waiting for app: TestExipryInsert to get deployed
2021/12/09 10:54:01 Waiting for app: TestExipryInsert to get deployed
2021/12/09 10:54:06 Waiting for app: TestExipryInsert to get deployed
2021/12/09 10:54:06 App: TestExipryInsert got deployed
2021/12/09 10:54:06 Starting bucket ops to source bucket
2021/12/09 10:54:51 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 10:54:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:54:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:54:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:54:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:54:56 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 10:55:01 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 10:55:07 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 10:55:12 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 10:55:12 App: TestExipryInsert got un-deployed
2021/12/09 10:55:12 No eventing-con process running
2021/12/09 10:55:12 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/09 10:55:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:55:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:55:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:55:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:55:25 Testing on the destination bucket for expiry insert
2021/12/09 10:55:25 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:55:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/09 10:55:32 Waiting for app: TestExipryInsert to get deployed
2021/12/09 10:55:37 Waiting for app: TestExipryInsert to get deployed
2021/12/09 10:55:42 Waiting for app: TestExipryInsert to get deployed
2021/12/09 10:55:47 Waiting for app: TestExipryInsert to get deployed
2021/12/09 10:55:47 App: TestExipryInsert got deployed
2021/12/09 10:55:47 Starting bucket ops to source bucket
2021/12/09 10:56:32 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 10:56:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:56:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:56:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:56:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:56:37 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 10:56:42 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 10:56:47 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 10:56:52 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 10:56:52 App: TestExipryInsert got un-deployed
2021/12/09 10:56:52 No eventing-con process running
2021/12/09 10:56:52 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/09 10:56:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:57:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:57:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 10:57:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:57:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:57:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.63s)
=== RUN   TestExipryReplace
2021/12/09 10:57:15 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:57:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 10:57:22 Waiting for app: TestExipryReplace to get deployed
2021/12/09 10:57:27 Waiting for app: TestExipryReplace to get deployed
2021/12/09 10:57:32 Waiting for app: TestExipryReplace to get deployed
2021/12/09 10:57:37 Waiting for app: TestExipryReplace to get deployed
2021/12/09 10:57:37 App: TestExipryReplace got deployed
2021/12/09 10:57:37 Starting bucket ops to source bucket
2021/12/09 10:58:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 10:58:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:58:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 10:58:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 10:58:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 10:58:27 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 10:58:32 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 10:58:37 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 10:58:42 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 10:58:42 App: TestExipryReplace got un-deployed
2021/12/09 10:58:42 No eventing-con process running
2021/12/09 10:58:42 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/09 10:58:44 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:58:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:58:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 10:58:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 10:58:56 Testing on the destination bucket for expiry insert
2021/12/09 10:58:56 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 10:58:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 10:59:02 Waiting for app: TestExipryReplace to get deployed
2021/12/09 10:59:07 Waiting for app: TestExipryReplace to get deployed
2021/12/09 10:59:12 Waiting for app: TestExipryReplace to get deployed
2021/12/09 10:59:17 Waiting for app: TestExipryReplace to get deployed
2021/12/09 10:59:17 App: TestExipryReplace got deployed
2021/12/09 10:59:17 Starting bucket ops to source bucket
2021/12/09 11:00:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:00:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:00:12 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:00:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:00:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:00:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 11:00:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:00:18 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 11:00:23 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 11:00:28 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 11:00:33 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 11:00:33 App: TestExipryReplace got un-deployed
2021/12/09 11:00:33 No eventing-con process running
2021/12/09 11:00:33 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/09 11:00:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:00:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:00:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:00:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:00:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (215.54s)
=== RUN   TestExipryUpsert
2021/12/09 11:00:51 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:00:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/09 11:00:57 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 11:01:02 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 11:01:07 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 11:01:12 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 11:01:12 App: TestExipryUpsert got deployed
2021/12/09 11:01:12 Starting bucket ops to source bucket
2021/12/09 11:01:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:02:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:02:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:02:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:02:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 11:02:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:02:08 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 11:02:13 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 11:02:18 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 11:02:23 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 11:02:23 App: TestExipryUpsert got un-deployed
2021/12/09 11:02:23 No eventing-con process running
2021/12/09 11:02:23 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/09 11:02:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:02:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:02:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:02:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:02:37 Testing on the destination bucket for expiry insert
2021/12/09 11:02:37 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:02:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/09 11:02:43 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 11:02:48 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 11:02:53 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 11:02:58 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 11:02:58 App: TestExipryUpsert got deployed
2021/12/09 11:02:58 Starting bucket ops to source bucket
2021/12/09 11:03:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:03:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:03:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:03:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:03:48 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 11:03:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:03:53 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 11:03:58 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 11:04:03 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 11:04:08 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 11:04:08 App: TestExipryUpsert got un-deployed
2021/12/09 11:04:08 No eventing-con process running
2021/12/09 11:04:08 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/09 11:04:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:04:20 Waiting for dst bucket item count to get to: 0 curr count: 53
2021/12/09 11:04:25 Waiting for dst bucket item count to get to: 0 curr count: 53
2021/12/09 11:04:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:04:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:04:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.55s)
=== RUN   TestCasReplace
2021/12/09 11:04:32 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:04:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 11:04:38 Waiting for app: TestCasReplace to get deployed
2021/12/09 11:04:43 Waiting for app: TestCasReplace to get deployed
2021/12/09 11:04:48 Waiting for app: TestCasReplace to get deployed
2021/12/09 11:04:53 Waiting for app: TestCasReplace to get deployed
2021/12/09 11:04:53 App: TestCasReplace got deployed
2021/12/09 11:04:53 Starting bucket ops to source bucket
2021/12/09 11:04:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:04:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:04:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 11:04:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:04:58 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 11:05:03 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 11:05:08 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 11:05:13 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 11:05:13 App: TestCasReplace got un-deployed
2021/12/09 11:05:13 No eventing-con process running
2021/12/09 11:05:13 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/09 11:05:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:05:25 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/09 11:05:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:05:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:05:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:05:32 Testing on the destination bucket for upsert with cas
2021/12/09 11:05:32 Starting bucket ops to source bucket
2021/12/09 11:05:32 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:05:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 11:05:38 Waiting for app: TestCasReplace to get deployed
2021/12/09 11:05:43 Waiting for app: TestCasReplace to get deployed
2021/12/09 11:05:48 Waiting for app: TestCasReplace to get deployed
2021/12/09 11:05:53 Waiting for app: TestCasReplace to get deployed
2021/12/09 11:05:53 App: TestCasReplace got deployed
2021/12/09 11:05:53 Starting bucket ops to source bucket
2021/12/09 11:05:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:06:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:06:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:06:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:06:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 11:06:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:06:08 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 11:06:13 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 11:06:18 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 11:06:23 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 11:06:23 App: TestCasReplace got un-deployed
2021/12/09 11:06:23 No eventing-con process running
2021/12/09 11:06:23 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/09 11:06:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:06:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:06:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:06:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:06:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.16s)
=== RUN   TestCasDelete
2021/12/09 11:06:42 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:06:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 11:06:48 Waiting for app: TestCasDelete to get deployed
2021/12/09 11:06:53 Waiting for app: TestCasDelete to get deployed
2021/12/09 11:06:58 Waiting for app: TestCasDelete to get deployed
2021/12/09 11:07:03 Waiting for app: TestCasDelete to get deployed
2021/12/09 11:07:03 App: TestCasDelete got deployed
2021/12/09 11:07:03 Starting bucket ops to source bucket
2021/12/09 11:07:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:07:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:07:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 11:07:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:07:08 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 11:07:13 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 11:07:18 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 11:07:23 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 11:07:23 App: TestCasDelete got un-deployed
2021/12/09 11:07:23 No eventing-con process running
2021/12/09 11:07:23 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/09 11:07:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:07:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:07:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:07:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:07:37 Testing on the destination bucket for delete with cas
2021/12/09 11:07:37 Starting bucket ops to source bucket
2021/12/09 11:07:37 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:07:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 11:07:43 Waiting for app: TestCasDelete to get deployed
2021/12/09 11:07:48 Waiting for app: TestCasDelete to get deployed
2021/12/09 11:07:53 Waiting for app: TestCasDelete to get deployed
2021/12/09 11:07:58 Waiting for app: TestCasDelete to get deployed
2021/12/09 11:07:58 App: TestCasDelete got deployed
2021/12/09 11:07:58 Starting bucket ops to source bucket
2021/12/09 11:08:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:08:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:08:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:08:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:08:09 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 11:08:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:08:14 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 11:08:19 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 11:08:24 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 11:08:29 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 11:08:29 App: TestCasDelete got un-deployed
2021/12/09 11:08:29 No eventing-con process running
2021/12/09 11:08:29 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/09 11:08:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:08:41 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/09 11:08:46 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/09 11:08:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:08:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:08:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.58s)
=== RUN   TestCountersIncrement
2021/12/09 11:08:52 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 11:08:58 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 11:08:58 Removing node(s): [] from the cluster
2021/12/09 11:09:03 Rebalance progress: 100
2021/12/09 11:09:03 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:09:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/09 11:09:10 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 11:09:15 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 11:09:20 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 11:09:25 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 11:09:25 App: TestCountersIncrement got deployed
2021/12/09 11:09:25 Starting bucket ops to source bucket
2021/12/09 11:09:31 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 11:09:36 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/09 11:09:36 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/09 11:09:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:09:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:09:41 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 11:09:46 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 11:09:51 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 11:09:56 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 11:09:56 App: TestCountersIncrement got un-deployed
2021/12/09 11:09:56 No eventing-con process running
2021/12/09 11:09:56 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/09 11:09:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:10:08 Waiting for dst bucket item count to get to: 0 curr count: 576
2021/12/09 11:10:13 Waiting for dst bucket item count to get to: 0 curr count: 63
2021/12/09 11:10:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:10:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:10:25 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 11:10:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:10:25 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:10:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/09 11:10:31 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 11:10:36 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 11:10:41 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 11:10:46 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 11:10:46 App: TestCountersIncrement got deployed
2021/12/09 11:10:46 Starting bucket ops to source bucket
2021/12/09 11:10:52 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/09 11:10:52 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/09 11:10:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:10:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:10:57 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 11:11:02 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 11:11:07 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 11:11:12 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 11:11:12 App: TestCountersIncrement got un-deployed
2021/12/09 11:11:12 No eventing-con process running
2021/12/09 11:11:12 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/09 11:11:14 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:11:24 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/09 11:11:29 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 11:11:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:11:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:11:41 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 11:11:46 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 11:11:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:11:46 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/09 11:11:51 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.47s)
=== RUN   TestCountersDecrement
2021/12/09 11:11:51 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 11:11:51 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 11:11:51 Removing node(s): [] from the cluster
2021/12/09 11:11:56 Rebalance progress: 100
2021/12/09 11:11:56 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:11:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/09 11:12:02 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 11:12:07 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 11:12:12 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 11:12:17 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 11:12:17 App: TestCountersDecrement got deployed
2021/12/09 11:12:17 Starting bucket ops to source bucket
2021/12/09 11:12:24 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 11:12:29 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/09 11:12:34 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/09 11:12:34 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/09 11:12:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:12:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:12:39 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 11:12:44 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 11:12:49 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 11:12:54 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 11:12:54 App: TestCountersDecrement got un-deployed
2021/12/09 11:12:54 No eventing-con process running
2021/12/09 11:12:54 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/09 11:12:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:13:05 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/09 11:13:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:13:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:13:17 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 11:13:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:13:17 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:13:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/09 11:13:23 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 11:13:28 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 11:13:33 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 11:13:39 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 11:13:39 App: TestCountersDecrement got deployed
2021/12/09 11:13:39 Starting bucket ops to source bucket
2021/12/09 11:13:45 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/09 11:13:50 Waiting for dst bucket item count to get to: 2048 curr count: 595
2021/12/09 11:13:55 Waiting for dst bucket item count to get to: 2048 curr count: 1619
2021/12/09 11:13:55 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/09 11:13:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:13:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:14:00 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 11:14:05 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 11:14:10 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 11:14:15 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 11:14:15 App: TestCountersDecrement got un-deployed
2021/12/09 11:14:15 No eventing-con process running
2021/12/09 11:14:15 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/09 11:14:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:14:27 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/09 11:14:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:14:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:14:38 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 11:14:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:14:38 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/09 11:14:44 Rebalance progress: 100
--- PASS: TestCountersDecrement (172.74s)
=== RUN   TestCacheGet
2021/12/09 11:14:44 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:14:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/09 11:14:50 Waiting for app: TestCacheGet to get deployed
2021/12/09 11:14:55 Waiting for app: TestCacheGet to get deployed
2021/12/09 11:15:00 Waiting for app: TestCacheGet to get deployed
2021/12/09 11:15:05 Waiting for app: TestCacheGet to get deployed
2021/12/09 11:15:05 App: TestCacheGet got deployed
2021/12/09 11:15:05 Starting bucket ops to source bucket
2021/12/09 11:15:10 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/09 11:15:15 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/09 11:15:20 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/09 11:15:25 Waiting for dst bucket item count to get to: 20 curr count: 4
2021/12/09 11:15:30 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/09 11:15:30 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/09 11:15:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:15:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:15:35 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 11:15:40 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 11:15:45 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 11:15:50 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 11:15:50 App: TestCacheGet got un-deployed
2021/12/09 11:15:50 No eventing-con process running
2021/12/09 11:15:50 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/09 11:15:52 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:16:02 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/09 11:16:07 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/09 11:16:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:16:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:16:19 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/09 11:16:24 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/09 11:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (100.30s)
=== RUN   TestCacheRYOW
2021/12/09 11:16:24 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:16:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/09 11:16:30 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 11:16:35 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 11:16:40 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 11:16:45 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 11:16:45 App: TestCacheRYOW got deployed
2021/12/09 11:16:45 Starting bucket ops to source bucket
2021/12/09 11:16:50 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/09 11:16:50 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/09 11:16:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:16:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:16:55 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 11:17:00 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 11:17:05 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 11:17:10 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 11:17:10 App: TestCacheRYOW got un-deployed
2021/12/09 11:17:10 No eventing-con process running
2021/12/09 11:17:10 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/09 11:17:12 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:17:22 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:17:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:17:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:17:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:17:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:17:44 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/09 11:17:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.10s)
=== RUN   TestCacheExpiry
2021/12/09 11:17:44 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:17:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/09 11:17:50 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 11:17:55 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 11:18:00 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 11:18:05 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 11:18:05 App: TestCacheExpiry got deployed
2021/12/09 11:18:05 Starting bucket ops to source bucket
2021/12/09 11:18:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 11:18:15 Waiting for dst bucket item count to get to: 100 curr count: 26
2021/12/09 11:18:20 Waiting for dst bucket item count to get to: 100 curr count: 26
2021/12/09 11:18:25 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/09 11:18:30 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/09 11:18:30 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 11:18:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:18:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:18:35 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 11:18:41 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 11:18:46 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 11:18:51 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 11:18:51 App: TestCacheExpiry got un-deployed
2021/12/09 11:18:51 No eventing-con process running
2021/12/09 11:18:51 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/09 11:18:52 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:19:02 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:19:07 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/09 11:19:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:19:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:19:19 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:19:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:19:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.20s)
=== RUN   TestCacheExpiryCustom
2021/12/09 11:19:24 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:19:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/09 11:19:30 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 11:19:35 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 11:19:40 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 11:19:45 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 11:19:45 App: TestCacheExpiryCustom got deployed
2021/12/09 11:19:45 Starting bucket ops to source bucket
2021/12/09 11:19:51 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 11:19:56 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/09 11:20:01 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/09 11:20:06 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/09 11:20:11 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/09 11:20:11 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 11:20:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:20:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:20:16 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 11:20:21 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 11:20:26 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 11:20:31 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 11:20:31 App: TestCacheExpiryCustom got un-deployed
2021/12/09 11:20:31 No eventing-con process running
2021/12/09 11:20:31 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/09 11:20:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:20:42 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:20:47 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/09 11:20:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:20:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:20:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:21:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:21:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.13s)
=== RUN   TestCacheOverflowCustom
2021/12/09 11:21:04 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:21:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/09 11:21:10 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 11:21:16 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 11:21:21 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 11:21:26 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 11:21:26 App: TestCacheOverflowCustom got deployed
2021/12/09 11:21:26 Starting bucket ops to source bucket
2021/12/09 11:21:31 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/09 11:21:36 Waiting for dst bucket item count to get to: 35 curr count: 13
2021/12/09 11:21:41 Waiting for dst bucket item count to get to: 35 curr count: 13
2021/12/09 11:21:46 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/09 11:21:51 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/09 11:21:51 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/09 11:21:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:21:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:21:56 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 11:22:01 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 11:22:06 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 11:22:11 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 11:22:11 App: TestCacheOverflowCustom got un-deployed
2021/12/09 11:22:11 No eventing-con process running
2021/12/09 11:22:11 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/09 11:22:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:22:23 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/09 11:22:28 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/09 11:22:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:22:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:22:39 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/09 11:22:44 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/09 11:22:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.15s)
=== RUN   TestStrictMode
2021/12/09 11:22:44 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:22:45 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/09 11:22:45 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:22:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/09 11:22:52 Waiting for app: TestDataTypes to get deployed
2021/12/09 11:22:57 Waiting for app: TestDataTypes to get deployed
2021/12/09 11:23:02 Waiting for app: TestDataTypes to get deployed
2021/12/09 11:23:07 Waiting for app: TestDataTypes to get deployed
2021/12/09 11:23:07 App: TestDataTypes got deployed
2021/12/09 11:23:07 Starting bucket ops to source bucket
2021/12/09 11:23:12 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/09 11:23:12 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/09 11:23:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:23:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:23:17 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 11:23:22 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 11:23:27 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 11:23:32 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 11:23:32 App: TestDataTypes got un-deployed
2021/12/09 11:23:32 No eventing-con process running
2021/12/09 11:23:32 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/09 11:23:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:23:44 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 11:23:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:23:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:23:55 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/09 11:23:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (70.04s)
=== RUN   TestEnoentGet
2021/12/09 11:23:55 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:23:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/09 11:24:02 Waiting for app: TestEnoentGet to get deployed
2021/12/09 11:24:07 Waiting for app: TestEnoentGet to get deployed
2021/12/09 11:24:12 Waiting for app: TestEnoentGet to get deployed
2021/12/09 11:24:17 Waiting for app: TestEnoentGet to get deployed
2021/12/09 11:24:17 App: TestEnoentGet got deployed
2021/12/09 11:24:17 Starting bucket ops to source bucket
2021/12/09 11:24:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 11:24:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 11:24:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:24:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:24:27 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 11:24:32 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 11:24:37 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 11:24:42 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 11:24:42 App: TestEnoentGet got un-deployed
2021/12/09 11:24:42 No eventing-con process running
2021/12/09 11:24:42 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/09 11:24:44 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:24:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:24:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:24:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:25:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:25:11 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/09 11:25:16 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/09 11:25:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:25:16 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:25:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/09 11:25:22 Waiting for app: TestEnoentGet to get deployed
2021/12/09 11:25:27 Waiting for app: TestEnoentGet to get deployed
2021/12/09 11:25:32 Waiting for app: TestEnoentGet to get deployed
2021/12/09 11:25:37 Waiting for app: TestEnoentGet to get deployed
2021/12/09 11:25:37 App: TestEnoentGet got deployed
2021/12/09 11:25:37 Starting bucket ops to source bucket
2021/12/09 11:25:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 11:25:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 11:25:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:25:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:25:47 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 11:25:52 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 11:25:57 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 11:26:02 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 11:26:02 App: TestEnoentGet got un-deployed
2021/12/09 11:26:02 No eventing-con process running
2021/12/09 11:26:02 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/09 11:26:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:26:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:26:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:26:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:26:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:26:31 Waiting for dst bucket item count to get to: 0 curr count: 86
2021/12/09 11:26:36 Waiting for dst bucket item count to get to: 0 curr count: 86
2021/12/09 11:26:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.24s)
=== RUN   TestEnoentDelete
2021/12/09 11:26:36 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:26:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 11:26:42 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 11:26:47 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 11:26:52 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 11:26:57 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 11:26:57 App: TestEnoentDelete got deployed
2021/12/09 11:26:57 Starting bucket ops to source bucket
2021/12/09 11:27:02 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 11:27:02 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 11:27:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:27:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:27:07 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 11:27:12 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 11:27:17 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 11:27:22 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 11:27:22 App: TestEnoentDelete got un-deployed
2021/12/09 11:27:22 No eventing-con process running
2021/12/09 11:27:22 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/09 11:27:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:27:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:27:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:27:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:27:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:27:51 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/09 11:27:56 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/09 11:27:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:27:56 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:27:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 11:28:02 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 11:28:07 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 11:28:12 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 11:28:17 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 11:28:17 App: TestEnoentDelete got deployed
2021/12/09 11:28:17 Starting bucket ops to source bucket
2021/12/09 11:28:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 11:28:27 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/09 11:28:32 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/09 11:28:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 11:28:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:28:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:28:37 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 11:28:42 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 11:28:47 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 11:28:52 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 11:28:52 App: TestEnoentDelete got un-deployed
2021/12/09 11:28:52 No eventing-con process running
2021/12/09 11:28:53 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/09 11:28:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:29:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:29:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:29:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:29:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:29:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:29:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:29:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (170.30s)
=== RUN   TestError
2021/12/09 11:29:36 Waiting for app: TestError to get un-deployed
2021/12/09 11:29:36 App: TestError got un-deployed
2021/12/09 11:29:36 No eventing-con process running
2021/12/09 11:29:36 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/09 11:29:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:29:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:29:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:29:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:29:50 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:29:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/09 11:29:56 Waiting for app: TestError to get deployed
2021/12/09 11:30:01 Waiting for app: TestError to get deployed
2021/12/09 11:30:06 Waiting for app: TestError to get deployed
2021/12/09 11:30:11 Waiting for app: TestError to get deployed
2021/12/09 11:30:11 App: TestError got deployed
2021/12/09 11:30:11 Starting bucket ops to source bucket
2021/12/09 11:30:16 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/09 11:30:21 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/09 11:30:21 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/09 11:30:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:30:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:30:26 Waiting for app: TestError to get un-deployed
2021/12/09 11:30:31 Waiting for app: TestError to get un-deployed
2021/12/09 11:30:36 Waiting for app: TestError to get un-deployed
2021/12/09 11:30:41 Waiting for app: TestError to get un-deployed
2021/12/09 11:30:41 App: TestError got un-deployed
2021/12/09 11:30:41 No eventing-con process running
2021/12/09 11:30:41 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/09 11:30:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:30:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:30:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 11:30:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:31:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:31:10 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/09 11:31:15 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/09 11:31:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.68s)
=== RUN   TestCRLF
2021/12/09 11:31:25 Waiting for app: TestCRLF to get un-deployed
2021/12/09 11:31:25 App: TestCRLF got un-deployed
2021/12/09 11:31:25 No eventing-con process running
2021/12/09 11:31:25 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/09 11:31:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:31:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:31:33 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:31:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:31:38 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:31:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/09 11:31:44 Waiting for app: TestCRLF to get deployed
2021/12/09 11:31:49 Waiting for app: TestCRLF to get deployed
2021/12/09 11:31:54 Waiting for app: TestCRLF to get deployed
2021/12/09 11:31:59 Waiting for app: TestCRLF to get deployed
2021/12/09 11:31:59 App: TestCRLF got deployed
2021/12/09 11:31:59 Starting bucket ops to source bucket
2021/12/09 11:32:05 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 11:32:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 11:32:15 Waiting for dst bucket item count to get to: 100 curr count: 40
2021/12/09 11:32:15 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 11:32:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:32:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:32:20 Waiting for app: TestCRLF to get un-deployed
2021/12/09 11:32:25 Waiting for app: TestCRLF to get un-deployed
2021/12/09 11:32:30 Waiting for app: TestCRLF to get un-deployed
2021/12/09 11:32:35 Waiting for app: TestCRLF to get un-deployed
2021/12/09 11:32:35 App: TestCRLF got un-deployed
2021/12/09 11:32:35 No eventing-con process running
2021/12/09 11:32:35 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/09 11:32:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:32:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 11:32:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:32:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:32:58 Waiting for dst bucket item count to get to: 0 curr count: 81
2021/12/09 11:33:03 Waiting for dst bucket item count to get to: 0 curr count: 21
2021/12/09 11:33:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.67s)
=== RUN   TestImportExport
2021/12/09 11:33:13 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 11:33:13 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/09 11:33:13 No eventing-con process running
2021/12/09 11:33:13 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/09 11:33:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:33:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:33:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:33:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:33:32 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 11:33:32 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/09 11:33:32 No eventing-con process running
2021/12/09 11:33:32 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/09 11:33:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:33:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:33:40 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:33:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:33:46 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:33:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 11:33:47 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:33:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 11:33:53 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 11:33:58 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 11:34:03 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 11:34:08 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 11:34:08 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/09 11:34:13 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/09 11:34:18 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/09 11:34:23 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/09 11:34:23 App: TestImportExport_bucket_op_on_update got deployed
2021/12/09 11:34:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:34:28 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 11:34:33 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 11:34:38 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 11:34:43 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 11:34:43 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/09 11:34:43 No eventing-con process running
2021/12/09 11:34:43 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/09 11:34:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:34:48 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 11:34:53 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 11:34:58 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 11:35:03 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 11:35:03 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/09 11:35:03 No eventing-con process running
2021/12/09 11:35:03 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/09 11:35:21 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 11:35:21 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/09 11:35:21 No eventing-con process running
2021/12/09 11:35:21 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/09 11:35:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:35:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:35:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:35:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:35:39 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 11:35:39 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/09 11:35:39 No eventing-con process running
2021/12/09 11:35:39 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/09 11:35:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:35:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:35:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:35:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.39s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/09 11:36:03 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:36:03 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 11:36:03 No eventing-con process running
2021/12/09 11:36:03 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/09 11:36:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:36:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:36:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:36:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:36:16 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:36:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 11:36:17 Starting bucket ops to source bucket
2021/12/09 11:36:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:36:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:36:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:36:40 Waiting for dst bucket item count to get to: 5000 curr count: 903
2021/12/09 11:36:45 Waiting for dst bucket item count to get to: 5000 curr count: 903
2021/12/09 11:36:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:36:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 11:36:50 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 11:36:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:36:51 Undeploying app: bucket_op_on_update
2021/12/09 11:36:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:36:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:37:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:37:06 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:37:11 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:37:11 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 11:37:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:37:11 No eventing-con process running
2021/12/09 11:37:12 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:37:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:37:55 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:37:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 11:37:56 Starting bucket ops to source bucket
2021/12/09 11:38:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:38:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:38:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:38:19 Waiting for dst bucket item count to get to: 5000 curr count: 2490
2021/12/09 11:38:24 Waiting for dst bucket item count to get to: 5000 curr count: 2490
2021/12/09 11:38:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:38:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 11:38:29 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 11:38:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:38:29 Undeploying app: bucket_op_on_update
2021/12/09 11:38:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:38:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:38:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:38:44 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:38:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:38:49 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 11:38:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:38:49 No eventing-con process running
2021/12/09 11:38:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:38:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:39:33 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:39:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 11:39:34 Starting bucket ops to source bucket
2021/12/09 11:39:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:39:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:39:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:39:57 Waiting for dst bucket item count to get to: 5000 curr count: 1531
2021/12/09 11:40:02 Waiting for dst bucket item count to get to: 5000 curr count: 4031
2021/12/09 11:40:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:40:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 11:40:07 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 11:40:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:40:07 Undeploying app: bucket_op_on_update
2021/12/09 11:40:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:40:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:40:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:40:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:40:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:40:27 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 11:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:40:27 No eventing-con process running
2021/12/09 11:40:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:40:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:41:11 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:41:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 11:41:12 Starting bucket ops to source bucket
2021/12/09 11:41:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:41:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:41:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:41:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:41:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:41:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 11:41:40 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 11:41:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:41:40 Undeploying app: bucket_op_on_update
2021/12/09 11:41:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:41:46 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:41:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:41:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:42:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:42:01 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 11:42:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:42:01 No eventing-con process running
2021/12/09 11:42:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:42:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:42:45 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:42:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 11:42:46 Starting bucket ops to source bucket
2021/12/09 11:42:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:42:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:43:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:43:09 Waiting for dst bucket item count to get to: 5000 curr count: 2704
2021/12/09 11:43:14 Waiting for dst bucket item count to get to: 5000 curr count: 2704
2021/12/09 11:43:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:43:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 11:43:19 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 11:43:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:43:19 Undeploying app: bucket_op_on_update
2021/12/09 11:43:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:43:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:43:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:43:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:43:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 11:43:39 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 11:43:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:43:39 No eventing-con process running
2021/12/09 11:43:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:43:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:44:23 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.12s)
=== RUN   TestOnUpdateN1QLOp
2021/12/09 11:44:33 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 11:44:33 App: TestOnUpdateN1QLOp got un-deployed
2021/12/09 11:44:33 No eventing-con process running
2021/12/09 11:44:33 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/09 11:44:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:44:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:44:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:44:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:44:46 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:44:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/09 11:44:48 Starting bucket ops to source bucket
2021/12/09 11:44:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:45:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:45:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:45:11 Waiting for dst bucket item count to get to: 5000 curr count: 91
2021/12/09 11:45:16 Waiting for dst bucket item count to get to: 5000 curr count: 91
2021/12/09 11:45:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:45:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:45:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:45:21 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 11:45:26 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 11:45:31 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 11:45:36 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 11:45:36 App: TestOnUpdateN1QLOp got un-deployed
2021/12/09 11:45:36 No eventing-con process running
2021/12/09 11:45:36 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/09 11:45:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:45:48 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 11:45:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:45:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:45:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 11:46:04 Waiting for dst bucket item count to get to: 0 curr count: 2517
2021/12/09 11:46:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.39s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/09 11:46:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 11:46:14 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/09 11:46:14 No eventing-con process running
2021/12/09 11:46:14 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/09 11:46:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:46:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:46:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:46:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:46:28 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:46:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 11:46:29 Starting bucket ops to source bucket
2021/12/09 11:46:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:46:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:46:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:46:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:46:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/09 11:46:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:46:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:46:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:47:02 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 11:47:07 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 11:47:12 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 11:47:17 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 11:47:17 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/09 11:47:17 No eventing-con process running
2021/12/09 11:47:17 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/09 11:47:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:47:29 Waiting for dst bucket item count to get to: 0 curr count: 4747
2021/12/09 11:47:34 Waiting for dst bucket item count to get to: 0 curr count: 2247
2021/12/09 11:47:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:47:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:47:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 11:47:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.21s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/09 11:47:55 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 11:47:55 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 11:47:56 No eventing-con process running
2021/12/09 11:47:56 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/09 11:47:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:48:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:48:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:48:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:48:09 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:48:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 11:48:10 Starting bucket ops to source bucket
2021/12/09 11:48:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:48:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:48:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:48:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:48:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:48:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:48:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:48:38 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 11:48:43 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 11:48:48 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 11:48:53 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 11:48:53 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 11:48:53 No eventing-con process running
2021/12/09 11:48:53 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/09 11:48:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:49:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 11:49:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:49:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:49:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 11:49:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (91.52s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/09 11:49:27 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 11:49:27 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/09 11:49:27 No eventing-con process running
2021/12/09 11:49:27 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/09 11:49:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:49:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:49:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:49:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:49:41 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:49:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/09 11:49:42 Starting bucket ops to source bucket
2021/12/09 11:49:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:49:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:50:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:50:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:50:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:50:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:50:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:50:10 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 11:50:15 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 11:50:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 11:50:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 11:50:25 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/09 11:50:25 No eventing-con process running
2021/12/09 11:50:25 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/09 11:50:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:50:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 11:50:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:50:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:50:49 Waiting for dst bucket item count to get to: 0 curr count: 4455
2021/12/09 11:50:54 Waiting for dst bucket item count to get to: 0 curr count: 1955
2021/12/09 11:50:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (96.53s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/09 11:51:04 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 11:51:04 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/09 11:51:04 No eventing-con process running
2021/12/09 11:51:04 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/09 11:51:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:51:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:51:12 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:51:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:51:17 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:51:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/09 11:51:19 Starting bucket ops to source bucket
2021/12/09 11:51:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:51:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:51:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:51:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:51:47 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/09 11:51:47 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:51:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:51:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:51:52 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 11:51:57 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 11:52:02 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 11:52:07 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 11:52:07 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/09 11:52:07 No eventing-con process running
2021/12/09 11:52:07 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/09 11:52:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:52:19 Waiting for dst bucket item count to get to: 0 curr count: 4737
2021/12/09 11:52:24 Waiting for dst bucket item count to get to: 0 curr count: 2237
2021/12/09 11:52:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:52:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:52:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 11:52:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.02s)
=== RUN   TestOnDeleteBucketOp
2021/12/09 11:52:46 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 11:52:46 App: TestOnDeleteBucketOp got un-deployed
2021/12/09 11:52:46 No eventing-con process running
2021/12/09 11:52:46 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/09 11:52:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:52:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:52:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:52:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:52:59 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:53:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 11:53:00 Starting bucket ops to source bucket
2021/12/09 11:53:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:53:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:53:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:53:21 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:53:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:53:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:53:26 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 11:53:31 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 11:53:36 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 11:53:41 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 11:53:41 App: TestOnDeleteBucketOp got un-deployed
2021/12/09 11:53:41 No eventing-con process running
2021/12/09 11:53:41 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/09 11:53:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:53:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:53:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:54:00 Waiting for dst bucket item count to get to: 0 curr count: 3353
2021/12/09 11:54:05 Waiting for dst bucket item count to get to: 0 curr count: 3353
2021/12/09 11:54:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (89.83s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/09 11:54:15 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 11:54:15 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/09 11:54:15 No eventing-con process running
2021/12/09 11:54:15 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/09 11:54:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:54:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:54:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:54:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:54:29 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:54:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/09 11:54:30 Starting bucket ops to source bucket
2021/12/09 11:54:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:54:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:54:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:54:56 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/09 11:54:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:54:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:54:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:55:01 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 11:55:06 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 11:55:11 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 11:55:16 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 11:55:16 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/09 11:55:16 No eventing-con process running
2021/12/09 11:55:16 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/09 11:55:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:55:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:55:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:55:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 11:55:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.62s)
=== RUN   TestTimerBucketOp
2021/12/09 11:55:45 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 11:55:45 App: TestTimerBucketOp got un-deployed
2021/12/09 11:55:45 No eventing-con process running
2021/12/09 11:55:45 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/09 11:55:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:55:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:55:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:55:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:55:59 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:56:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 11:56:00 Starting bucket ops to source bucket
2021/12/09 11:56:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:56:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:56:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:56:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:56:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:56:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:56:38 Waiting for dst bucket item count to get to: 5000 curr count: 1168
2021/12/09 11:56:43 Waiting for dst bucket item count to get to: 5000 curr count: 3668
2021/12/09 11:56:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:56:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:56:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:56:48 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 11:56:53 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 11:56:58 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 11:57:03 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 11:57:03 App: TestTimerBucketOp got un-deployed
2021/12/09 11:57:03 No eventing-con process running
2021/12/09 11:57:03 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/09 11:57:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:57:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 11:57:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:57:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:57:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 11:57:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (111.38s)
=== RUN   TestTimerInPastBucketOp
2021/12/09 11:57:36 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 11:57:36 App: TestTimerInPastBucketOp got un-deployed
2021/12/09 11:57:36 No eventing-con process running
2021/12/09 11:57:36 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/09 11:57:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:57:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:57:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:57:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:57:50 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:57:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 11:57:51 Starting bucket ops to source bucket
2021/12/09 11:57:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:58:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:58:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:58:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:58:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:58:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 11:58:29 Waiting for dst bucket item count to get to: 5000 curr count: 2221
2021/12/09 11:58:34 Waiting for dst bucket item count to get to: 5000 curr count: 2221
2021/12/09 11:58:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 11:58:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 11:58:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 11:58:39 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 11:58:44 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 11:58:49 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 11:58:54 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 11:58:54 App: TestTimerInPastBucketOp got un-deployed
2021/12/09 11:58:54 No eventing-con process running
2021/12/09 11:58:54 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/09 11:58:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:59:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 11:59:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:59:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:59:18 Waiting for dst bucket item count to get to: 0 curr count: 3347
2021/12/09 11:59:23 Waiting for dst bucket item count to get to: 0 curr count: 847
2021/12/09 11:59:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (116.49s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/09 11:59:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 11:59:33 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 11:59:33 No eventing-con process running
2021/12/09 11:59:33 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/09 11:59:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:59:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:59:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 11:59:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 11:59:46 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 11:59:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 11:59:48 Starting bucket ops to source bucket
2021/12/09 11:59:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:00:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:00:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:00:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:00:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:00:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:00:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:00:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:00:31 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 12:00:31 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 12:00:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:00:31 Undeploying app: bucket_op_with_timer
2021/12/09 12:00:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:00:36 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:00:41 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:00:46 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:00:51 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:00:51 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 12:00:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:00:51 No eventing-con process running
2021/12/09 12:00:52 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:01:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:01:35 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:01:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 12:01:36 Starting bucket ops to source bucket
2021/12/09 12:01:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:01:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:01:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:01:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:02:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:02:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:02:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:02:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:02:19 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 12:02:19 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 12:02:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:02:19 Undeploying app: bucket_op_with_timer
2021/12/09 12:02:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:02:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:02:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:02:34 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:02:39 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:02:39 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 12:02:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:02:39 No eventing-con process running
2021/12/09 12:02:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:02:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:03:23 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:03:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 12:03:24 Starting bucket ops to source bucket
2021/12/09 12:03:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:03:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:03:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:03:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:03:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:03:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:04:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:04:07 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/09 12:04:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:04:12 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 12:04:12 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 12:04:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:04:12 Undeploying app: bucket_op_with_timer
2021/12/09 12:04:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:04:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:04:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:04:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:04:32 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:04:32 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 12:04:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:04:32 No eventing-con process running
2021/12/09 12:04:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:04:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:05:16 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:05:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 12:05:17 Starting bucket ops to source bucket
2021/12/09 12:05:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:05:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:05:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:05:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:05:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:05:50 Waiting for dst bucket item count to get to: 5000 curr count: 1317
2021/12/09 12:05:55 Waiting for dst bucket item count to get to: 5000 curr count: 1317
2021/12/09 12:06:00 Waiting for dst bucket item count to get to: 5000 curr count: 4338
2021/12/09 12:06:05 Waiting for dst bucket item count to get to: 5000 curr count: 4338
2021/12/09 12:06:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:06:10 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 12:06:10 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 12:06:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:06:10 Undeploying app: bucket_op_with_timer
2021/12/09 12:06:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:06:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:06:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:06:25 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:06:30 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:06:30 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 12:06:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:06:30 No eventing-con process running
2021/12/09 12:06:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:06:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:07:14 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:07:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 12:07:16 Starting bucket ops to source bucket
2021/12/09 12:07:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:07:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:07:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:07:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:07:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:07:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:07:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:07:59 Waiting for dst bucket item count to get to: 5000 curr count: 4370
2021/12/09 12:08:04 Waiting for dst bucket item count to get to: 5000 curr count: 4370
2021/12/09 12:08:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:08:09 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 12:08:09 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 12:08:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:08:09 Undeploying app: bucket_op_with_timer
2021/12/09 12:08:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:08:14 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:08:19 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:08:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:08:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 12:08:29 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 12:08:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:08:29 No eventing-con process running
2021/12/09 12:08:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:08:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:09:13 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (589.90s)
=== RUN   TestMultipleHandlers
2021/12/09 12:09:23 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 12:09:23 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/09 12:09:23 No eventing-con process running
2021/12/09 12:09:23 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/09 12:09:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:09:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:09:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:09:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:09:41 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 12:09:41 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/09 12:09:41 No eventing-con process running
2021/12/09 12:09:41 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/09 12:09:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:09:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:09:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:09:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:09:55 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:09:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 12:09:56 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:09:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/09 12:10:02 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 12:10:07 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 12:10:12 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 12:10:17 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 12:10:17 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/09 12:10:22 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/09 12:10:27 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/09 12:10:32 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/09 12:10:32 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/09 12:10:32 Starting bucket ops to source bucket
2021/12/09 12:10:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:10:47 Waiting for dst bucket item count to get to: 5000 curr count: 4972
2021/12/09 12:10:52 Waiting for dst bucket item count to get to: 5000 curr count: 4972
2021/12/09 12:10:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:10:52 Starting bucket ops to source bucket
2021/12/09 12:11:07 Waiting for dst bucket item count to get to: 10000 curr count: 6067
2021/12/09 12:11:12 Waiting for dst bucket item count to get to: 10000 curr count: 6067
2021/12/09 12:11:12 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 12:11:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:11:12 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:11:13 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:11:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:11:18 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 12:11:23 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 12:11:28 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 12:11:33 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 12:11:33 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/09 12:11:33 No eventing-con process running
2021/12/09 12:11:33 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/09 12:11:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:11:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:11:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:11:52 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 12:11:57 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 12:11:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:11:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:12:02 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 12:12:07 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 12:12:12 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 12:12:17 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 12:12:17 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/09 12:12:17 No eventing-con process running
2021/12/09 12:12:17 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/09 12:12:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:12:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:12:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:12:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (197.73s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/09 12:12:40 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 12:12:41 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/09 12:12:41 No eventing-con process running
2021/12/09 12:12:41 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/09 12:12:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:12:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:12:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:12:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:12:54 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:12:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 12:12:55 Starting bucket ops to source bucket
2021/12/09 12:13:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:13:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:13:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:13:18 Waiting for dst bucket item count to get to: 5000 curr count: 606
2021/12/09 12:13:23 Waiting for dst bucket item count to get to: 5000 curr count: 3106
2021/12/09 12:13:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:13:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:13:28 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 12:13:28 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 12:13:28 Pausing app: bucket_op_on_update
2021/12/09 12:13:28 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:13:33 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 12:13:33 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 12:13:33 Resuming app: bucket_op_on_update
2021/12/09 12:13:34 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:13:34 Starting bucket ops to source bucket
2021/12/09 12:13:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:13:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:13:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:13:52 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 12:13:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:13:57 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 12:13:57 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 12:13:57 Pausing app: bucket_op_on_update
2021/12/09 12:13:58 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:14:03 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 12:14:03 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 12:14:03 Resuming app: bucket_op_on_update
2021/12/09 12:14:04 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:14:04 Starting bucket ops to source bucket
2021/12/09 12:14:12 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 12:14:17 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 12:14:22 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 12:14:22 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/09 12:14:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:14:27 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 12:14:27 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 12:14:27 Pausing app: bucket_op_on_update
2021/12/09 12:14:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:14:32 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 12:14:32 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 12:14:32 Resuming app: bucket_op_on_update
2021/12/09 12:14:33 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:14:33 Starting bucket ops to source bucket
2021/12/09 12:14:41 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 12:14:46 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 12:14:51 Waiting for dst bucket item count to get to: 20000 curr count: 15415
2021/12/09 12:14:56 Waiting for dst bucket item count to get to: 20000 curr count: 15415
2021/12/09 12:14:56 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/09 12:14:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:15:01 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 12:15:01 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 12:15:01 Pausing app: bucket_op_on_update
2021/12/09 12:15:01 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:15:06 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 12:15:06 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 12:15:06 Resuming app: bucket_op_on_update
2021/12/09 12:15:07 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:15:07 Starting bucket ops to source bucket
2021/12/09 12:15:15 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 12:15:20 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 12:15:25 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 12:15:25 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/09 12:15:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:15:30 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 12:15:30 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 12:15:30 Pausing app: bucket_op_on_update
2021/12/09 12:15:30 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:15:35 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 12:15:35 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 12:15:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:15:40 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 12:15:46 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 12:15:51 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 12:15:56 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 12:15:56 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/09 12:15:56 No eventing-con process running
2021/12/09 12:15:56 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/09 12:15:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:16:07 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/09 12:16:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:16:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:16:19 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/09 12:16:24 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/09 12:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.62s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/09 12:16:34 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 12:16:34 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/09 12:16:34 No eventing-con process running
2021/12/09 12:16:34 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/09 12:16:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:16:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:16:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:16:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:16:48 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:16:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 12:16:49 Starting bucket ops to source bucket
2021/12/09 12:16:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:17:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:17:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:17:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:17:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:17:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:17:27 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/09 12:17:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:17:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:17:32 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 12:17:32 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 12:17:32 Pausing app: bucket_op_with_timer
2021/12/09 12:17:32 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:17:37 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 12:17:37 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 12:17:37 Resuming app: bucket_op_with_timer
2021/12/09 12:17:38 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:17:38 Starting bucket ops to source bucket
2021/12/09 12:17:46 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:17:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:17:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:18:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:18:06 Waiting for dst bucket item count to get to: 10000 curr count: 5445
2021/12/09 12:18:11 Waiting for dst bucket item count to get to: 10000 curr count: 6139
2021/12/09 12:18:16 Waiting for dst bucket item count to get to: 10000 curr count: 8194
2021/12/09 12:18:16 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 12:18:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:18:21 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 12:18:21 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 12:18:21 Pausing app: bucket_op_with_timer
2021/12/09 12:18:21 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:18:26 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 12:18:26 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 12:18:26 Resuming app: bucket_op_with_timer
2021/12/09 12:18:27 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:18:27 Starting bucket ops to source bucket
2021/12/09 12:18:35 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 12:18:40 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 12:18:45 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 12:18:50 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 12:18:55 Waiting for dst bucket item count to get to: 15000 curr count: 11977
2021/12/09 12:19:00 Waiting for dst bucket item count to get to: 15000 curr count: 11977
2021/12/09 12:19:00 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/09 12:19:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:19:05 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 12:19:05 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 12:19:05 Pausing app: bucket_op_with_timer
2021/12/09 12:19:06 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:19:11 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 12:19:11 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 12:19:11 Resuming app: bucket_op_with_timer
2021/12/09 12:19:12 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:19:12 Starting bucket ops to source bucket
2021/12/09 12:19:20 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 12:19:25 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 12:19:30 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 12:19:35 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 12:19:40 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 12:19:45 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 12:19:45 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/09 12:19:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:19:50 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 12:19:50 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 12:19:50 Pausing app: bucket_op_with_timer
2021/12/09 12:19:50 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:19:55 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 12:19:55 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 12:19:55 Resuming app: bucket_op_with_timer
2021/12/09 12:19:56 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:19:56 Starting bucket ops to source bucket
2021/12/09 12:20:04 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 12:20:09 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 12:20:14 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 12:20:19 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 12:20:24 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 12:20:29 Waiting for dst bucket item count to get to: 25000 curr count: 22538
2021/12/09 12:20:34 Waiting for dst bucket item count to get to: 25000 curr count: 22538
2021/12/09 12:20:34 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/09 12:20:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:20:39 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 12:20:39 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 12:20:39 Pausing app: bucket_op_with_timer
2021/12/09 12:20:39 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:20:44 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 12:20:44 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 12:20:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:20:49 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 12:20:54 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 12:20:59 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 12:21:04 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 12:21:04 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/09 12:21:04 No eventing-con process running
2021/12/09 12:21:04 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/09 12:21:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:21:16 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/09 12:21:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:21:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:21:28 Waiting for dst bucket item count to get to: 0 curr count: 15267
2021/12/09 12:21:33 Waiting for dst bucket item count to get to: 0 curr count: 2767
2021/12/09 12:21:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (308.66s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/09 12:21:43 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 12:21:43 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/09 12:21:43 No eventing-con process running
2021/12/09 12:21:43 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/09 12:21:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:21:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:21:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:21:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:21:56 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:21:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/09 12:22:02 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 12:22:07 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 12:22:12 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 12:22:17 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 12:22:17 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/09 12:22:17 Starting bucket ops to source bucket
2021/12/09 12:22:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:22:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:22:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:22:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:22:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:22:52 Waiting for dst bucket item count to get to: 5000 curr count: 2501
2021/12/09 12:22:57 Waiting for dst bucket item count to get to: 5000 curr count: 2501
2021/12/09 12:22:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:22:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:22:57 Pausing app: bucket_op_with_timer
2021/12/09 12:22:57 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:23:02 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/09 12:23:02 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/09 12:23:02 Resuming app: bucket_op_with_timer
2021/12/09 12:23:03 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:23:03 Starting bucket ops to source bucket
2021/12/09 12:23:15 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:23:20 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:23:25 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:23:30 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:23:35 Waiting for dst bucket item count to get to: 10000 curr count: 6672
2021/12/09 12:23:40 Waiting for dst bucket item count to get to: 10000 curr count: 9172
2021/12/09 12:23:40 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 12:23:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:23:45 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 12:23:50 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 12:23:55 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 12:24:00 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 12:24:00 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/09 12:24:00 No eventing-con process running
2021/12/09 12:24:00 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/09 12:24:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:24:12 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 12:24:17 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 12:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:24:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:24:29 Waiting for dst bucket item count to get to: 0 curr count: 9265
2021/12/09 12:24:34 Waiting for dst bucket item count to get to: 0 curr count: 4265
2021/12/09 12:24:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (180.92s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/09 12:24:44 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 12:24:44 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/09 12:24:44 No eventing-con process running
2021/12/09 12:24:44 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/09 12:24:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:24:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:24:52 Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:24:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:24:57 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:24:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/09 12:25:03 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 12:25:08 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 12:25:13 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 12:25:18 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 12:25:18 App: TestPauseResumeWithEventingReb got deployed
2021/12/09 12:25:18 Starting bucket ops to source bucket
2021/12/09 12:25:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:25:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:25:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:25:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:25:48 Waiting for dst bucket item count to get to: 5000 curr count: 1222
2021/12/09 12:25:53 Waiting for dst bucket item count to get to: 5000 curr count: 3153
2021/12/09 12:25:58 Waiting for dst bucket item count to get to: 5000 curr count: 4431
2021/12/09 12:25:58 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:25:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:25:58 Pausing app: bucket_op_with_timer
2021/12/09 12:25:58 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:26:03 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/09 12:26:03 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/09 12:26:03 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 12:26:04 Removing node(s): [] from the cluster
2021/12/09 12:26:09 Rebalance progress: 29.16666666666666
2021/12/09 12:26:14 Rebalance progress: 29.16666666666666
2021/12/09 12:26:19 Rebalance progress: 100
2021/12/09 12:26:19 Resuming app: bucket_op_with_timer
2021/12/09 12:26:20 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:26:20 Starting bucket ops to source bucket
2021/12/09 12:26:32 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:26:37 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:26:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:26:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:26:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 12:26:57 Waiting for dst bucket item count to get to: 10000 curr count: 7176
2021/12/09 12:27:02 Waiting for dst bucket item count to get to: 10000 curr count: 9676
2021/12/09 12:27:02 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 12:27:02 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 12:27:07 Rebalance progress: 29.16666666666666
2021/12/09 12:27:12 Rebalance progress: 29.16666666666666
2021/12/09 12:27:17 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/09 12:27:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 12:27:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 12:27:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 12:28:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 12:28:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 12:28:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 12:28:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 12:29:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 12:29:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 12:29:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 12:29:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 64d140eda24ed780133b8ce7b0addf94
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
Producer node: 172.31.2.134:9300	Node UUID: 19946832b30b089b3a53c097b9fa3252
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
Producer node: 127.0.0.1:9303	Node UUID: 0772d037dcdfa40489cd17d98c3495e3
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/09 12:30:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:30:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:30:12 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 12:30:17 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 12:30:22 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 12:30:27 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 12:30:27 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/09 12:30:27 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:30:27 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/09 12:30:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:30:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 12:30:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 12:30:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:30:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:30:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 12:30:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (382.04s)
=== RUN   TestBucketDeleteAfterPause
2021/12/09 12:31:06 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/09 12:31:06 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/09 12:31:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:31:06 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/09 12:31:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:31:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:31:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:31:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:31:19 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:31:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/09 12:31:25 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 12:31:30 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 12:31:35 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 12:31:40 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 12:31:40 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/09 12:31:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:31:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:31:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:31:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:31:50 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/09 12:31:50 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:31:55 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/09 12:31:55 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/09 12:31:55 handler_test.go:716: Deleting source bucket: default
2021/12/09 12:32:01 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/09 12:32:01 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/09 12:32:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/09 12:32:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/09 12:32:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:32:07 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/09 12:32:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:32:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 12:32:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:32:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:32:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 12:32:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 12:32:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (98.94s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/09 12:32:45 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 12:32:45 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/09 12:32:45 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:32:45 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/09 12:32:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:32:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:32:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:32:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:32:58 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:32:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/09 12:33:04 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 12:33:09 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 12:33:14 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 12:33:19 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 12:33:19 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/09 12:33:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:33:54 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/09 12:33:54 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:33:59 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/09 12:33:59 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/09 12:36:59 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/09 12:36:59 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/09 12:37:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/09 12:37:05 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/09 12:37:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:37:05 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/09 12:37:10 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/09 12:37:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:37:10 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3114 expected 5000
2021/12/09 12:37:15 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/09 12:37:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:37:15 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 4234 expected 5000
2021/12/09 12:37:20 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/09 12:37:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:37:20 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/09 12:37:20 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/09 12:37:20 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:37:30 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 12:37:36 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 12:37:41 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 12:37:41 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/09 12:37:41 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:37:41 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/09 12:37:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:37:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 12:37:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 12:37:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:37:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:38:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.84s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/09 12:38:14 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 12:38:14 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/09 12:38:14 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:38:14 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/09 12:38:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:38:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:38:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:38:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:38:27 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:38:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 12:38:33 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 12:38:38 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 12:38:43 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 12:38:48 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 12:38:48 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/09 12:38:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:38:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:39:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 859
2021/12/09 12:39:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 859
2021/12/09 12:39:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:39:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:39:07 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/09 12:39:07 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:39:37 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/09 12:39:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 12:39:43 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 12:39:48 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 12:39:53 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 12:39:58 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 12:39:58 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/09 12:39:58 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:40:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 12:40:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 12:40:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:40:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:40:16 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/09 12:40:16 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:40:26 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 12:40:31 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 12:40:36 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 12:40:36 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/09 12:40:36 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:40:36 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/09 12:40:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:40:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:40:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:40:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.46s)
=== RUN   TestCPPWorkerCleanup
2021/12/09 12:40:59 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 12:40:59 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/09 12:40:59 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:40:59 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/09 12:41:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:41:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:41:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:41:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:41:12 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:41:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/09 12:41:13 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:41:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:41:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:41:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:41:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 404
2021/12/09 12:41:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2358
2021/12/09 12:41:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4454
2021/12/09 12:41:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:41:45 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:41:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:41:51 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 12:41:56 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 12:42:01 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 12:42:06 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 12:42:06 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/09 12:42:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:42:06 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/09 12:42:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:42:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 12:42:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:42:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:42:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4487
2021/12/09 12:42:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1987
2021/12/09 12:42:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (134.61s)
=== RUN   TestWithUserXattrs
2021/12/09 12:43:14 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 12:43:14 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/09 12:43:14 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:43:14 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/09 12:43:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:43:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:43:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:43:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:43:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:43:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:43:31 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:43:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/09 12:43:37 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 12:43:42 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 12:43:47 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 12:43:52 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 12:43:52 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/09 12:43:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:44:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:44:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:44:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:44:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:44:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:44:11 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 12:44:16 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 12:44:21 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 12:44:26 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 12:44:26 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/09 12:44:26 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:44:26 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/09 12:44:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:44:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 12:44:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:44:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:44:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 12:44:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 12:44:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (110.78s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/09 12:45:04 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 12:45:04 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/09 12:45:04 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:45:04 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/09 12:45:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:45:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:45:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:45:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:45:17 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:45:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/09 12:45:24 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 12:45:29 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 12:45:34 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 12:45:39 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 12:45:39 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/09 12:45:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:45:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 12:45:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 313
2021/12/09 12:45:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2813
2021/12/09 12:45:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 12:45:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:46:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:46:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3879
2021/12/09 12:46:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3879
2021/12/09 12:46:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:46:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:46:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4900
2021/12/09 12:46:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2400
2021/12/09 12:46:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:46:34 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:46:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 18322
2021/12/09 12:47:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 18322
2021/12/09 12:47:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/09 12:47:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:47:04 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:47:09 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 12:47:14 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 12:47:19 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 12:47:24 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 12:47:24 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/09 12:47:24 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:47:29 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 12:47:29 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/09 12:47:29 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:47:29 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/09 12:47:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:47:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 15014
2021/12/09 12:47:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 15014
2021/12/09 12:47:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:47:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:47:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/09 12:47:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (182.27s)
=== RUN   TestMetaBucketDelete
2021/12/09 12:48:07 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/09 12:48:07 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/09 12:48:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:48:07 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/09 12:48:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:48:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:48:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:48:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:48:20 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:48:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 12:48:26 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 12:48:31 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 12:48:36 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 12:48:41 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 12:48:41 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/09 12:48:41 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/09 12:48:42 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/09 12:48:47 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/09 12:48:47 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/09 12:49:02 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/09 12:49:02 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/09 12:49:02 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:49:02 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/09 12:49:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:49:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:49:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:49:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.62s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/09 12:49:25 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 12:49:25 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 12:49:25 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:49:25 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/09 12:49:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:49:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:49:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:49:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:49:38 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:49:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 12:49:58 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/09 12:49:59 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/09 12:50:14 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 12:50:14 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 12:50:29 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 12:50:29 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 12:50:29 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:50:29 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/09 12:50:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:50:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:50:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:50:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.70s)
=== RUN   TestSourceBucketDelete
2021/12/09 12:50:52 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/09 12:50:52 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/09 12:50:52 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:50:52 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/09 12:50:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:50:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:51:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:51:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:51:05 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:51:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 12:51:11 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 12:51:16 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 12:51:21 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 12:51:26 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 12:51:26 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/09 12:51:26 handler_test.go:945: Deleting source bucket: default
2021/12/09 12:51:28 handler_test.go:947: Deleted source bucket: default
2021/12/09 12:51:33 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/09 12:51:33 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/09 12:51:48 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/09 12:51:48 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/09 12:51:48 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:51:48 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/09 12:51:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:51:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:51:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:52:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (78.88s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/09 12:52:11 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/09 12:52:11 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/09 12:52:11 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:52:11 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/09 12:52:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:52:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:52:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:52:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:52:24 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:52:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 12:52:44 handler_test.go:966: Deleting source bucket: default
2021/12/09 12:52:45 handler_test.go:968: Deleted source bucket: default
2021/12/09 12:53:00 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/09 12:53:00 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/09 12:53:16 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/09 12:53:16 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/09 12:53:16 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:53:16 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/09 12:53:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:53:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:53:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:53:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.88s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/09 12:53:39 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 12:53:39 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 12:53:39 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:53:39 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/09 12:53:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:53:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:53:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:53:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:53:52 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:53:53 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 12:54:12 handler_test.go:988: Deleting source bucket: default
2021/12/09 12:54:13 handler_test.go:990: Deleted source bucket: default
2021/12/09 12:54:13 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/09 12:54:14 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/09 12:54:29 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 12:54:29 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 12:54:44 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 12:54:44 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 12:54:44 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:54:44 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/09 12:54:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:54:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:54:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:54:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.29s)
=== RUN   TestUndeployDuringBootstrap
2021/12/09 12:55:07 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 12:55:07 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/09 12:55:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:55:07 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/09 12:55:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:55:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:55:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:55:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:55:20 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:55:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 12:55:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:55:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 12:55:33 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:55:33 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/09 12:55:33 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/09 12:55:33 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/09 12:55:38 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/09 12:55:43 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 12:55:48 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 12:55:53 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 12:55:53 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/09 12:55:53 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:56:18 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 12:56:18 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/09 12:56:18 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:56:18 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/09 12:56:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:56:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 12:56:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 12:56:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:56:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:56:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2863
2021/12/09 12:56:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.11s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/09 12:56:51 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:56:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/09 12:56:57 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 12:57:02 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 12:57:07 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 12:57:12 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 12:57:12 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/09 12:57:12 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/09 12:57:12 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 12:57:13 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/09 12:57:18 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/09 12:57:18 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/09 12:57:18 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:57:18 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/09 12:57:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:57:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:57:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:57:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.49s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/09 12:57:41 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 12:57:41 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/09 12:57:41 eventingSetup.go:46: No eventing-con process running
2021/12/09 12:57:41 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/09 12:57:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:57:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:57:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 12:57:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 12:57:54 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/09 12:57:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/09 12:58:00 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 12:58:05 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 12:58:10 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 12:58:15 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 12:58:15 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/09 12:58:15 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 12:58:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 12:58:50 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 12:58:55 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 12:59:00 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 12:59:05 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 12:59:05 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/09 12:59:05 eventingSetup.go:46: No eventing-con process running
2021/12/09 13:00:50 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 13:00:50 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/09 13:00:50 eventingSetup.go:46: No eventing-con process running
2021/12/09 13:00:50 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/09 13:00:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:01:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 13:01:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 13:01:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:01:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:01:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4330
2021/12/09 13:01:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1830
2021/12/09 13:01:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (232.92s)
=== RUN   TestUndeployWithKVFailover
2021/12/09 13:01:34 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/09 13:01:34 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/09 13:01:34 eventingSetup.go:46: No eventing-con process running
2021/12/09 13:01:34 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/09 13:01:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:01:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:01:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:01:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:01:47 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:01:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/09 13:01:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 13:01:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:02:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:02:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:02:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:02:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:02:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:02:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:02:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 13:02:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 13:02:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 13:02:30 Rebalance progress: 29.16666666666666
2021/12/09 13:02:35 Rebalance progress: 29.16666666666666
2021/12/09 13:02:40 Rebalance progress: 100
2021/12/09 13:02:40 Failing over: [127.0.0.1:19003]
2021/12/09 13:02:50 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 13:03:50 Removing node(s): [] from the cluster
2021/12/09 13:03:55 Rebalance progress: 29.16666666666666
2021/12/09 13:04:00 Rebalance progress: 29.16666666666666
2021/12/09 13:04:05 Rebalance progress: 100
2021/12/09 13:04:10 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/09 13:04:10 App: TestUndeployWithKVFailover got un-deployed
2021/12/09 13:04:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:04:15 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/09 13:04:15 App: TestUndeployWithKVFailover got un-deployed
2021/12/09 13:04:15 No eventing-con process running
2021/12/09 13:04:15 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/09 13:04:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:04:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 13:04:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:04:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:04:39 Waiting for dst bucket item count to get to: 0 curr count: 4388
2021/12/09 13:04:44 Waiting for dst bucket item count to get to: 0 curr count: 1888
2021/12/09 13:04:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (200.07s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/09 13:04:54 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 13:04:54 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/09 13:04:54 No eventing-con process running
2021/12/09 13:04:54 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/09 13:04:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:05:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:05:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:05:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:05:07 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:05:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/09 13:05:13 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 13:05:18 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 13:05:23 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 13:05:28 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 13:05:28 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/09 13:05:28 Starting bucket ops to source bucket
2021/12/09 13:05:48 Waiting for dst bucket item count to get to: 20000 curr count: 13201
2021/12/09 13:05:53 Waiting for dst bucket item count to get to: 20000 curr count: 17287
2021/12/09 13:05:53 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/09 13:05:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:05:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:06:07 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/09 13:06:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:06:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:06:18 Waiting for dst bucket item count to get to: 0 curr count: 16536
2021/12/09 13:06:23 Waiting for dst bucket item count to get to: 0 curr count: 6536
2021/12/09 13:06:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:06:23 Starting bucket ops to source bucket
2021/12/09 13:06:36 Waiting for dst bucket item count to get to: 10000 curr count: 2494
2021/12/09 13:06:41 Waiting for dst bucket item count to get to: 10000 curr count: 6358
2021/12/09 13:06:46 Waiting for dst bucket item count to get to: 10000 curr count: 8864
2021/12/09 13:06:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 13:06:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:06:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:06:51 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 13:06:56 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 13:07:01 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 13:07:06 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 13:07:06 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/09 13:07:06 No eventing-con process running
2021/12/09 13:07:06 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/09 13:07:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:07:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 13:07:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:07:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:07:29 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 13:07:34 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 13:07:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (170.61s)
=== RUN   TestTimerOverwrite
2021/12/09 13:07:44 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 13:07:44 App: TestTimerOverwrite got un-deployed
2021/12/09 13:07:44 No eventing-con process running
2021/12/09 13:07:44 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/09 13:07:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:07:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:07:52 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:07:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:07:57 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:07:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/09 13:08:03 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 13:08:08 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 13:08:13 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 13:08:18 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 13:08:18 App: TestTimerOverwrite got deployed
2021/12/09 13:08:28 Starting bucket ops to source bucket
2021/12/09 13:08:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:08:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:08:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:08:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:08:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:09:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:09:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:09:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:09:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:09:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:09:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:09:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:09:38 Waiting for dst bucket item count to get to: 5000 curr count: 2507
2021/12/09 13:09:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 13:09:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:09:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:09:53 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 13:09:58 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 13:10:03 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 13:10:08 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 13:10:08 App: TestTimerOverwrite got un-deployed
2021/12/09 13:10:08 No eventing-con process running
2021/12/09 13:10:08 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/09 13:10:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:10:19 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 13:10:24 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 13:10:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:10:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:10:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 13:10:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (181.94s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/09 13:10:41 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 13:10:41 Removing node(s): [] from the cluster
2021/12/09 13:10:46 Rebalance progress: 100
2021/12/09 13:10:46 Failing over: [127.0.0.1:19003]
2021/12/09 13:10:51 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/09 13:10:51 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/09 13:10:51 No eventing-con process running
2021/12/09 13:10:51 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/09 13:10:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:10:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:11:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:11:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:11:05 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:11:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 13:11:06 Starting bucket ops to source bucket
2021/12/09 13:11:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:11:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:11:16 Context: Post to set retry counter, response code: 200 dump: 
2021/12/09 13:11:16 set retry counter, response &{[] }
2021/12/09 13:12:21 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/09 13:12:21 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/09 13:12:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:12:21 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/09 13:12:26 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/09 13:12:26 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/09 13:12:26 No eventing-con process running
2021/12/09 13:12:26 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/09 13:12:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:12:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 13:12:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:12:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:12:49 Waiting for dst bucket item count to get to: 0 curr count: 2697
2021/12/09 13:12:54 Waiting for dst bucket item count to get to: 0 curr count: 197
2021/12/09 13:12:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:12:54 Removing node(s): [] from the cluster
2021/12/09 13:12:59 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (142.90s)
=== RUN   TestOnUpdateSrcMutation
2021/12/09 13:13:09 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 13:13:09 App: TestOnUpdateSrcMutation got un-deployed
2021/12/09 13:13:09 No eventing-con process running
2021/12/09 13:13:09 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/09 13:13:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:13:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:13:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:13:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:13:22 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:13:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/09 13:13:23 Starting bucket ops to source bucket
2021/12/09 13:13:45 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 13:13:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:13:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:13:50 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 13:13:55 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 13:14:00 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 13:14:05 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 13:14:05 App: TestOnUpdateSrcMutation got un-deployed
2021/12/09 13:14:05 No eventing-con process running
2021/12/09 13:14:05 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/09 13:14:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:14:17 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 13:14:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:14:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:14:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (84.38s)
=== RUN   TestOnDeleteSrcMutation
2021/12/09 13:14:33 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 13:14:33 App: TestOnDeleteSrcMutation got un-deployed
2021/12/09 13:14:33 No eventing-con process running
2021/12/09 13:14:33 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/09 13:14:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:14:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:14:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:14:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:14:46 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:14:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/09 13:14:48 Starting bucket ops to source bucket
2021/12/09 13:15:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 13:15:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:15:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:15:21 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 13:15:26 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 13:15:31 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 13:15:36 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 13:15:36 App: TestOnDeleteSrcMutation got un-deployed
2021/12/09 13:15:36 No eventing-con process running
2021/12/09 13:15:36 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/09 13:15:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:15:48 Waiting for dst bucket item count to get to: 0 curr count: 2769
2021/12/09 13:15:53 Waiting for dst bucket item count to get to: 0 curr count: 269
2021/12/09 13:15:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:15:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:16:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.12s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/09 13:16:10 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 13:16:10 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/09 13:16:10 No eventing-con process running
2021/12/09 13:16:10 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/09 13:16:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:16:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:16:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:16:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:16:23 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:16:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 13:16:24 Starting bucket ops to source bucket
2021/12/09 13:17:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 13:17:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:17:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:17:11 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 13:17:16 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 13:17:21 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 13:17:26 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 13:17:26 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/09 13:17:26 No eventing-con process running
2021/12/09 13:17:26 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/09 13:17:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:17:37 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 13:17:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:17:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:17:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (104.49s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/09 13:17:54 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 13:17:54 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/09 13:17:54 No eventing-con process running
2021/12/09 13:17:54 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/09 13:17:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:18:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:18:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:18:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:18:07 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:18:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 13:18:08 Starting bucket ops to source bucket
2021/12/09 13:18:47 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 13:18:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:18:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:18:52 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 13:18:57 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 13:19:02 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 13:19:07 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 13:19:07 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/09 13:19:07 No eventing-con process running
2021/12/09 13:19:07 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/09 13:19:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:19:19 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 13:19:24 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 13:19:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:19:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:19:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (106.28s)
=== RUN   TestInterHandlerRecursion
2021/12/09 13:19:40 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 13:19:40 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 13:19:40 No eventing-con process running
2021/12/09 13:19:40 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/09 13:19:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:19:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:19:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:19:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:19:58 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 13:19:58 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 13:19:58 No eventing-con process running
2021/12/09 13:19:58 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/09 13:20:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:20:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:20:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:20:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:20:12 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:20:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/09 13:20:13 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/09 13:20:18 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 13:20:23 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 13:20:28 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 13:20:33 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 13:20:33 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/09 13:20:33 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:20:33 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/09 13:20:33 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/09 13:20:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:20:38 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 13:20:43 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 13:20:48 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 13:20:53 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 13:20:53 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 13:20:53 No eventing-con process running
2021/12/09 13:20:53 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/09 13:20:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:20:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:21:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:21:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:21:11 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 13:21:11 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 13:21:11 No eventing-con process running
2021/12/09 13:21:11 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/09 13:21:12 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:21:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:21:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:21:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.64s)
=== RUN   TestInterBucketRecursion
2021/12/09 13:21:29 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:21:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 13:21:30 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 13:21:30 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:21:30 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/09 13:21:35 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 13:21:40 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 13:21:45 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 13:21:50 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 13:21:50 App: TestInterBucketRecursion_function1 got deployed
2021/12/09 13:21:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:21:55 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 13:22:00 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 13:22:05 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 13:22:10 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 13:22:10 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/09 13:22:10 No eventing-con process running
2021/12/09 13:22:10 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/09 13:22:12 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:22:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:22:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:22:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:22:28 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/09 13:22:28 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/09 13:22:28 No eventing-con process running
2021/12/09 13:22:28 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/09 13:22:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:22:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:22:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:22:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.34s)
=== RUN   TestLargeHandler
2021/12/09 13:22:41 Changed force_compress value to false
2021/12/09 13:22:41 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:22:41 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/09 13:22:41 Changed force_compress value to true
2021/12/09 13:22:42 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:22:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/09 13:22:48 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 13:22:53 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 13:22:58 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 13:23:03 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 13:23:03 App: TestLargeHandler_function got deployed
2021/12/09 13:23:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:23:09 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 13:23:14 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 13:23:19 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 13:23:24 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 13:23:24 App: TestLargeHandler_function got un-deployed
2021/12/09 13:23:24 No eventing-con process running
2021/12/09 13:23:24 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/09 13:23:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:23:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:23:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:23:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.34s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/09 13:23:47 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 13:23:47 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 13:23:47 No eventing-con process running
2021/12/09 13:23:47 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/09 13:23:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:23:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:23:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:24:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:24:05 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 13:24:05 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 13:24:05 No eventing-con process running
2021/12/09 13:24:05 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/09 13:24:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:24:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:24:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:24:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:24:18 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:24:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/09 13:24:19 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/09 13:24:24 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 13:24:29 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 13:24:34 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 13:24:39 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 13:24:39 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/09 13:24:39 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:24:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/09 13:24:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/09 13:24:45 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 13:24:50 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 13:24:55 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 13:25:00 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 13:25:00 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/09 13:25:00 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/09 13:25:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:25:05 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 13:25:10 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 13:25:15 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 13:25:20 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 13:25:20 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 13:25:20 No eventing-con process running
2021/12/09 13:25:21 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/09 13:25:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:25:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:25:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:25:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:25:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:25:40 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 13:25:45 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 13:25:50 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 13:25:55 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 13:25:55 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 13:25:55 No eventing-con process running
2021/12/09 13:25:55 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/09 13:25:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:26:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:26:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:26:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.29s)
=== RUN   TestAllowInterBucketRecursion
2021/12/09 13:26:13 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:26:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 13:26:14 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 13:26:19 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 13:26:24 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 13:26:29 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 13:26:34 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 13:26:34 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/09 13:26:34 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:26:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/09 13:26:35 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/09 13:26:40 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 13:26:45 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 13:26:50 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 13:26:55 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 13:26:55 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/09 13:26:55 TestAllowInterBucketRecursion_function2 is deployed
2021/12/09 13:26:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:27:00 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 13:27:05 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 13:27:10 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 13:27:15 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 13:27:15 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/09 13:27:15 No eventing-con process running
2021/12/09 13:27:16 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/09 13:27:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:27:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:27:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:27:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:27:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:27:34 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 13:27:39 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 13:27:44 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 13:27:49 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 13:27:49 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/09 13:27:49 No eventing-con process running
2021/12/09 13:27:50 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/09 13:27:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:27:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:27:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:28:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (114.66s)
=== RUN   TestInterBucketRecursion2
2021/12/09 13:28:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:28:14 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:28:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:28:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:28:31 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:28:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 13:28:38 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 13:28:43 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 13:28:48 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 13:28:53 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 13:28:53 App: TestInterBucketRecursion2_function1 got deployed
2021/12/09 13:28:53 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:28:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/09 13:28:59 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 13:29:04 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 13:29:09 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 13:29:14 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 13:29:14 App: TestInterBucketRecursion2_function2 got deployed
2021/12/09 13:29:14 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:29:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/09 13:29:20 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 13:29:25 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 13:29:30 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 13:29:35 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 13:29:35 App: TestInterBucketRecursion2_function3 got deployed
2021/12/09 13:29:35 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:29:35 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/09 13:29:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:29:40 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 13:29:45 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 13:29:50 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 13:29:55 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 13:29:55 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/09 13:29:55 No eventing-con process running
2021/12/09 13:29:55 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/09 13:29:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:30:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:30:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:30:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:30:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:30:13 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 13:30:18 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 13:30:23 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 13:30:28 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 13:30:29 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/09 13:30:29 No eventing-con process running
2021/12/09 13:30:29 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/09 13:30:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:30:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:30:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:30:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:30:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:30:47 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 13:30:52 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 13:30:57 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 13:31:02 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 13:31:02 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/09 13:31:02 No eventing-con process running
2021/12/09 13:31:02 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/09 13:31:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:31:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:31:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:31:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:31:20 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/09 13:31:20 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/09 13:31:20 No eventing-con process running
2021/12/09 13:31:20 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/09 13:31:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:31:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:31:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:31:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (212.71s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/09 13:31:40 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 13:31:45 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/09 13:31:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 13:32:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 13:32:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 13:32:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 13:32:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 13:33:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 13:33:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 13:33:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 13:33:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/09 13:34:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:34:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 13:34:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 13:34:09 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/09 13:34:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:34:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 13:34:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 13:34:15 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/09 13:34:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:34:20 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/09 13:34:20 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/09 13:34:20 eventingSetup.go:46: No eventing-con process running
2021/12/09 13:34:20 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/09 13:34:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:34:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:34:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:34:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:34:38 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:34:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 13:34:44 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 13:34:49 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 13:34:54 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 13:34:59 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 13:34:59 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 64d140eda24ed780133b8ce7b0addf94
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 19946832b30b089b3a53c097b9fa3252
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 0772d037dcdfa40489cd17d98c3495e3
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/09 13:34:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:34:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 13:35:04 Rebalance progress: 29.16666666666666
2021/12/09 13:35:09 Rebalance progress: 29.16666666666666
2021/12/09 13:35:14 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 64d140eda24ed780133b8ce7b0addf94
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 19946832b30b089b3a53c097b9fa3252
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 0772d037dcdfa40489cd17d98c3495e3
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/09 13:35:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:35:14 handler_test.go:1674: Deleting source bucket: default
2021/12/09 13:35:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 13:35:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 13:35:21 Rebalance progress: 29.16666666666666
2021/12/09 13:35:26 Rebalance progress: 29.16666666666666
2021/12/09 13:35:31 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 905
	vb list dump: [0, 2-16, 18-21, 23-27, 29-34, 36-50, 52-62, 65-66, 69-73, 75-86, 88-91, 93-122, 125-132, 134-141, 143-144, 146-154, 156-157, 159-163, 165, 167-183, 186-208, 210-214, 216-240, 242-244, 246-252, 254, 257-270, 273-293, 295-302, 304-309, 311-316, 319-333, 335-339, 342, 344-352, 354-359, 363-376, 378-384, 386, 388-390, 392-393, 395-409, 411-419, 421-422, 424-437, 439-455, 457-460, 462-463, 466-467, 469-471, 473-474, 476, 478-487, 489-501, 503-505, 507-508, 511-571, 573-577, 579, 581-586, 588-607, 609-610, 612-615, 617-622, 624-648, 650-652, 655-665, 667-668, 670-687, 689-702, 704-706, 708-709, 711-720, 722-728, 730-741, 743-747, 749-775, 777, 780-784, 786-788, 790-800, 802-803, 805-807, 809-840, 842-843, 845-854, 856-858, 860, 862-876, 878-884, 886-888, 891-895, 897-923, 925-929, 931-953, 955-961, 963-972, 974, 976-981, 983-992, 996-1015, 1017-1018, 1020-1023]
	stream status: stopped
	len: 119
	vb list dump: [1, 17, 22, 28, 35, 51, 63-64, 67-68, 74, 87, 92, 123-124, 133, 142, 145, 155, 158, 164, 166, 184-185, 209, 215, 241, 245, 253, 255-256, 271-272, 294, 303, 310, 317-318, 334, 340-341, 343, 353, 360-362, 377, 385, 387, 391, 394, 410, 420, 423, 438, 456, 461, 464-465, 468, 472, 475, 477, 488, 502, 506, 509-510, 572, 578, 580, 587, 608, 611, 616, 623, 649, 653-654, 666, 669, 688, 703, 707, 710, 721, 729, 742, 748, 776, 778-779, 785, 789, 801, 804, 808, 841, 844, 855, 859, 861, 877, 885, 889-890, 896, 924, 930, 954, 962, 973, 975, 982, 993-995, 1016, 1019]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 64d140eda24ed780133b8ce7b0addf94
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0, 2-16, 18-21, 23-27, 29-34, 36-50, 52-62, 65-66, 69-73, 75-86, 88-91, 93-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-122, 125-132, 134-141, 143-144, 146-154, 156-157, 159-163, 165, 167-183, 186-208, 210-214, 216-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 99
	v: [228-240, 242-244, 246-252, 254, 257-270, 273-293, 295-302, 304-309, 311-316, 319-333, 335-339]
Producer node: 172.31.2.134:9300	Node UUID: 19946832b30b089b3a53c097b9fa3252
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-923, 925-929, 931-953, 955-961, 963-972, 974, 976-981, 983-992, 996-1015, 1017-1018, 1020-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-687, 689-702, 704-706, 708-709, 711-720, 722-728, 730-741, 743-747, 749-775, 777, 780-784, 786-788, 790-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-800, 802-803, 805-807, 809-840, 842-843, 845-854, 856-858, 860, 862-876, 878-884, 886-888, 891-895, 897-910]
Producer node: 127.0.0.1:9303	Node UUID: 0772d037dcdfa40489cd17d98c3495e3
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [342, 344-352, 354-359, 363-376, 378-384, 386, 388-390, 392-393, 395-409, 411-419, 421-422, 424-437, 439-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [457-460, 462-463, 466-467, 469-471, 473-474, 476, 478-487, 489-501, 503-505, 507-508, 511-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570-571, 573-577, 579, 581-586, 588-607, 609-610, 612-615, 617-622, 624-648, 650-652, 655-665, 667-668, 670-682]


2021/12/09 13:35:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:35:36 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/09 13:35:36 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/09 13:35:41 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/09 13:35:41 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/09 13:35:41 eventingSetup.go:46: No eventing-con process running
2021/12/09 13:35:41 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/09 13:35:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:35:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:35:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:35:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:35:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/09 13:35:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 13:36:00 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (267.28s)
=== RUN   TestN1QLRecursion
2021/12/09 13:36:03 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 13:36:03 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/09 13:36:03 Removing node(s): [] from the cluster
2021/12/09 13:36:08 Rebalance progress: 100
2021/12/09 13:36:14 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:36:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:36:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:36:31 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:36:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 13:36:32 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 13:36:37 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 13:36:42 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 13:36:47 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 13:36:52 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 13:36:52 App: TestN1QLRecursion_function1 got deployed
2021/12/09 13:36:52 TestN1QLRecursion_function1 is deployed
2021/12/09 13:36:52 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:36:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/09 13:36:53 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/09 13:36:58 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 13:37:03 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 13:37:08 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 13:37:13 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 13:37:13 App: TestN1QLRecursion_function3 got deployed
2021/12/09 13:37:13 TestN1QLRecursion_function3 is deployed
2021/12/09 13:37:13 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:37:13 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/09 13:37:13 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/09 13:37:13 Success: TestN1QLRecursion_function2 is not deployed
2021/12/09 13:37:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:37:18 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 13:37:23 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 13:37:28 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 13:37:33 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 13:37:33 App: TestN1QLRecursion_function1 got un-deployed
2021/12/09 13:37:33 No eventing-con process running
2021/12/09 13:37:33 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/09 13:37:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:37:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:37:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:37:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:37:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:37:52 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 13:37:57 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 13:38:02 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 13:38:07 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 13:38:07 App: TestN1QLRecursion_function3 got un-deployed
2021/12/09 13:38:07 No eventing-con process running
2021/12/09 13:38:07 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/09 13:38:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:38:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:38:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:38:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:38:22 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 13:38:27 Rebalance progress: 100
--- PASS: TestN1QLRecursion (144.87s)
=== RUN   TestN1QLAllowRecursion
2021/12/09 13:38:27 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 13:38:28 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/09 13:38:28 Removing node(s): [] from the cluster
2021/12/09 13:38:33 Rebalance progress: 100
2021/12/09 13:38:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:38:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:38:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:38:56 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:38:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 13:38:57 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 13:39:02 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 13:39:07 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 13:39:12 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 13:39:17 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 13:39:17 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/09 13:39:17 TestN1QLAllowRecursion_function1 is deployed
2021/12/09 13:39:17 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:39:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/09 13:39:18 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/09 13:39:23 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 13:39:28 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 13:39:33 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 13:39:38 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 13:39:38 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/09 13:39:38 TestN1QLAllowRecursion_function3 is deployed
2021/12/09 13:39:38 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:39:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/09 13:39:39 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/09 13:39:44 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 13:39:49 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 13:39:54 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 13:39:59 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 13:39:59 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/09 13:40:00 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/09 13:40:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:40:05 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 13:40:10 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 13:40:15 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 13:40:20 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 13:40:20 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/09 13:40:20 No eventing-con process running
2021/12/09 13:40:20 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/09 13:40:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:40:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:40:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:40:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:40:38 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 13:40:43 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 13:40:48 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 13:40:53 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 13:40:53 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/09 13:40:54 No eventing-con process running
2021/12/09 13:40:54 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/09 13:40:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:41:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:41:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:41:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:41:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:41:12 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 13:41:17 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 13:41:22 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 13:41:27 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 13:41:27 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/09 13:41:27 No eventing-con process running
2021/12/09 13:41:27 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/09 13:41:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:41:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:41:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:41:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:41:42 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 13:41:48 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.13s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/09 13:41:48 Sleeping for some time
2021/12/09 13:41:58 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 13:41:58 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/09 13:41:58 No eventing-con process running
2021/12/09 13:41:58 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/09 13:41:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:42:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:42:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:42:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:42:11 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:42:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 13:42:12 Starting bucket ops to source bucket
2021/12/09 13:43:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:43:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 13:43:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 13:43:07 Starting bucket ops to source bucket
2021/12/09 13:43:08 Starting bucket ops to source bucket
2021/12/09 13:44:14 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/09 13:44:19 Waiting for dst bucket item count to get to: 7000 curr count: 6000
2021/12/09 13:44:19 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/09 13:44:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:44:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:44:24 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 13:44:29 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 13:44:34 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 13:44:39 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 13:44:39 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/09 13:44:39 No eventing-con process running
2021/12/09 13:44:39 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/09 13:44:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:44:51 Waiting for dst bucket item count to get to: 0 curr count: 838
2021/12/09 13:44:56 Waiting for dst bucket item count to get to: 0 curr count: 838
2021/12/09 13:44:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:44:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:45:07 Waiting for dst bucket item count to get to: 0 curr count: 3500
2021/12/09 13:45:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.84s)
=== RUN   TestTimerOverWriteSameReference
2021/12/09 13:45:12 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 13:45:12 App: TestTimerOverWriteSameReference got un-deployed
2021/12/09 13:45:12 No eventing-con process running
2021/12/09 13:45:12 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/09 13:45:14 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:45:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:45:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:45:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:45:26 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:45:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/09 13:45:32 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 13:45:37 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 13:45:42 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 13:45:47 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 13:45:47 App: TestTimerOverWriteSameReference got deployed
2021/12/09 13:45:57 Starting bucket ops to source bucket
2021/12/09 13:46:58 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/09 13:47:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:47:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:47:13 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 13:47:18 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 13:47:24 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 13:47:29 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 13:47:29 App: TestTimerOverWriteSameReference got un-deployed
2021/12/09 13:47:29 No eventing-con process running
2021/12/09 13:47:29 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/09 13:47:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:47:40 Waiting for dst bucket item count to get to: 0 curr count: 738
2021/12/09 13:47:45 Waiting for dst bucket item count to get to: 0 curr count: 738
2021/12/09 13:47:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:47:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:47:57 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 13:47:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.73s)
=== RUN   TestCancelTimerBucketop
2021/12/09 13:48:02 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 13:48:02 App: TestCancelTimerBucketop got un-deployed
2021/12/09 13:48:02 No eventing-con process running
2021/12/09 13:48:02 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/09 13:48:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:48:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:48:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:48:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:48:16 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:48:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/09 13:48:22 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 13:48:27 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 13:48:32 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 13:48:37 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 13:48:37 App: TestCancelTimerBucketop got deployed
2021/12/09 13:48:37 Starting bucket ops to source bucket
2021/12/09 13:48:44 Starting bucket ops to source bucket
2021/12/09 13:49:46 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 13:49:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:49:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:49:52 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 13:49:57 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 13:50:02 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 13:50:07 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 13:50:07 App: TestCancelTimerBucketop got un-deployed
2021/12/09 13:50:07 No eventing-con process running
2021/12/09 13:50:07 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/09 13:50:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:50:19 Waiting for dst bucket item count to get to: 0 curr count: 839
2021/12/09 13:50:24 Waiting for dst bucket item count to get to: 0 curr count: 339
2021/12/09 13:50:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:50:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:50:35 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 13:50:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.20s)
=== RUN   TestJSExpiryDate
2021/12/09 13:50:40 Starting bucket ops to source bucket
2021/12/09 13:50:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 13:50:53 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/09 13:50:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/09 13:51:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:51:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:51:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:51:30 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 13:51:35 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 13:51:40 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 13:51:45 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 13:51:45 App: TestJSExpiryDate got un-deployed
2021/12/09 13:51:45 No eventing-con process running
2021/12/09 13:51:45 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/09 13:51:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:51:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:51:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:51:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.64s)
=== RUN   TestBinaryDoc
2021/12/09 13:52:04 Starting bucket ops to source bucket
2021/12/09 13:52:04 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:52:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/09 13:52:05 Starting bucket ops to source bucket
2021/12/09 13:52:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:52:05 Deleting source bucket docs now
2021/12/09 13:52:05 Starting bucket ops to source bucket
2021/12/09 13:52:42 Waiting for dst bucket item count to get to: 32 curr count: 21
2021/12/09 13:52:47 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/09 13:52:52 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/09 13:52:52 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/09 13:52:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:52:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:52:58 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 13:53:03 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 13:53:08 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 13:53:13 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 13:53:13 App: TestBinaryDoc got un-deployed
2021/12/09 13:53:13 No eventing-con process running
2021/12/09 13:53:13 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/09 13:53:14 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:53:24 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/09 13:53:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:53:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:53:36 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/09 13:53:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.30s)
=== RUN   TestConstantBindings
2021/12/09 13:53:36 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:53:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/09 13:53:37 Starting bucket ops to source bucket
2021/12/09 13:53:43 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/09 13:53:48 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/09 13:53:53 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/09 13:53:58 Waiting for dst bucket item count to get to: 6000 curr count: 552
2021/12/09 13:54:03 Waiting for dst bucket item count to get to: 6000 curr count: 552
2021/12/09 13:54:03 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/09 13:54:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:54:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:54:08 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 13:54:13 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 13:54:18 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 13:54:23 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 13:54:23 App: TestConstantBindings got un-deployed
2021/12/09 13:54:23 No eventing-con process running
2021/12/09 13:54:23 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/09 13:54:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:54:35 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 13:54:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:54:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:54:47 Waiting for dst bucket item count to get to: 0 curr count: 6000
2021/12/09 13:54:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (70.51s)
=== RUN   TestCodeInsight
2021/12/09 13:54:52 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 13:54:52 App: TestCodeInsight got un-deployed
2021/12/09 13:54:52 No eventing-con process running
2021/12/09 13:54:52 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/09 13:54:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:54:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:55:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:55:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:55:05 Starting bucket ops to source bucket
2021/12/09 13:55:06 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:55:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/09 13:55:12 Waiting for app: TestCodeInsight to get deployed
2021/12/09 13:55:17 Waiting for app: TestCodeInsight to get deployed
2021/12/09 13:55:22 Waiting for app: TestCodeInsight to get deployed
2021/12/09 13:55:27 Waiting for app: TestCodeInsight to get deployed
2021/12/09 13:55:27 App: TestCodeInsight got deployed
2021/12/09 13:55:32 Waiting for dst bucket item count to get to: 1000 curr count: 81
2021/12/09 13:55:32 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 13:55:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:55:37 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 13:55:42 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 13:55:47 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 13:55:52 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 13:55:52 App: TestCodeInsight got un-deployed
2021/12/09 13:55:52 No eventing-con process running
2021/12/09 13:55:52 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/09 13:55:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:56:04 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 13:56:09 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 13:56:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:56:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:56:21 Waiting for dst bucket item count to get to: 0 curr count: 995
2021/12/09 13:56:26 Waiting for dst bucket item count to get to: 0 curr count: 995
2021/12/09 13:56:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.30s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.06s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12643.179s

Functional tests took 3 hours, 30 minutes and 46 seconds.

N1ql tests

2021/12/09 13:56:35 Started n1ql
2021/12/09 13:56:45 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/09 13:56:45 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/09 13:56:45 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 13:56:45 Removing node(s): [] from the cluster
2021/12/09 13:56:50 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/09 13:56:55 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/09 13:56:55 App: TestRecursiveMutationN1QL got un-deployed
2021/12/09 13:56:55 No eventing-con process running
2021/12/09 13:56:55 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/09 13:56:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:57:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:57:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:57:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:57:09 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:57:09 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.71s)
=== RUN   TestFlexReset1
2021/12/09 13:57:14 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 13:57:14 App: TestFlexReset1 got un-deployed
2021/12/09 13:57:14 No eventing-con process running
2021/12/09 13:57:14 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/09 13:57:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:57:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:57:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:57:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:57:28 Starting bucket ops to source bucket
2021/12/09 13:57:28 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:57:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/09 13:57:34 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 13:57:40 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 13:57:45 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 13:57:50 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 13:57:50 App: TestFlexReset1 got deployed
2021/12/09 13:57:55 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/09 13:57:55 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 13:57:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:57:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:58:00 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 13:58:05 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 13:58:10 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 13:58:15 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 13:58:15 App: TestFlexReset1 got un-deployed
2021/12/09 13:58:15 No eventing-con process running
2021/12/09 13:58:15 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/09 13:58:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:58:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 13:58:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:58:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:58:38 Waiting for dst bucket item count to get to: 0 curr count: 797
2021/12/09 13:58:43 Waiting for dst bucket item count to get to: 0 curr count: 297
2021/12/09 13:58:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.17s)
=== RUN   TestFlexReset2
2021/12/09 13:58:48 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 13:58:48 App: TestFlexReset2 got un-deployed
2021/12/09 13:58:48 No eventing-con process running
2021/12/09 13:58:48 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/09 13:58:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:58:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:58:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 13:59:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 13:59:02 Starting bucket ops to source bucket
2021/12/09 13:59:03 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/09 13:59:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/09 13:59:09 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 13:59:14 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 13:59:19 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 13:59:24 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 13:59:24 App: TestFlexReset2 got deployed
2021/12/09 13:59:29 Waiting for dst bucket item count to get to: 1000 curr count: 719
2021/12/09 13:59:29 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 13:59:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 13:59:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 13:59:34 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 13:59:39 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 13:59:44 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 13:59:49 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 13:59:49 App: TestFlexReset2 got un-deployed
2021/12/09 13:59:49 No eventing-con process running
2021/12/09 13:59:49 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/09 13:59:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:00:01 Waiting for dst bucket item count to get to: 0 curr count: 926
2021/12/09 14:00:06 Waiting for dst bucket item count to get to: 0 curr count: 926
2021/12/09 14:00:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:00:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:00:18 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 14:00:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (94.20s)
=== RUN   TestN1QLLabelledBreak
2021/12/09 14:00:23 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 14:00:23 App: TestN1QLLabelledBreak got un-deployed
2021/12/09 14:00:23 No eventing-con process running
2021/12/09 14:00:23 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/09 14:00:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:00:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:00:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:00:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:00:36 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:00:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/09 14:00:42 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 14:00:47 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 14:00:52 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 14:00:57 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 14:00:57 App: TestN1QLLabelledBreak got deployed
2021/12/09 14:00:57 Starting bucket ops to source bucket
2021/12/09 14:01:07 Waiting for dst bucket item count to get to: 10000 curr count: 136
2021/12/09 14:01:12 Waiting for dst bucket item count to get to: 10000 curr count: 136
2021/12/09 14:01:17 Waiting for dst bucket item count to get to: 10000 curr count: 136
2021/12/09 14:01:22 Waiting for dst bucket item count to get to: 10000 curr count: 3217
2021/12/09 14:01:27 Waiting for dst bucket item count to get to: 10000 curr count: 3217
2021/12/09 14:01:32 Waiting for dst bucket item count to get to: 10000 curr count: 7272
2021/12/09 14:01:37 Waiting for dst bucket item count to get to: 10000 curr count: 7272
2021/12/09 14:01:37 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 14:01:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 14:01:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:01:42 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 14:01:47 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 14:01:52 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 14:01:57 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 14:01:57 App: TestN1QLLabelledBreak got un-deployed
2021/12/09 14:01:57 No eventing-con process running
2021/12/09 14:01:57 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/09 14:01:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:02:09 Waiting for dst bucket item count to get to: 0 curr count: 4939
2021/12/09 14:02:14 Waiting for dst bucket item count to get to: 0 curr count: 2439
2021/12/09 14:02:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:02:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:02:26 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 14:02:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.20s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/09 14:02:31 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 14:02:31 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/09 14:02:31 No eventing-con process running
2021/12/09 14:02:31 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/09 14:02:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:02:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:02:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:02:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:02:44 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:02:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/09 14:02:50 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 14:02:55 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 14:03:00 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 14:03:05 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 14:03:05 App: TestN1QLUnlabelledBreak got deployed
2021/12/09 14:03:05 Starting bucket ops to source bucket
2021/12/09 14:03:15 Waiting for dst bucket item count to get to: 10000 curr count: 585
2021/12/09 14:03:20 Waiting for dst bucket item count to get to: 10000 curr count: 585
2021/12/09 14:03:25 Waiting for dst bucket item count to get to: 10000 curr count: 585
2021/12/09 14:03:30 Waiting for dst bucket item count to get to: 10000 curr count: 5790
2021/12/09 14:03:35 Waiting for dst bucket item count to get to: 10000 curr count: 5790
2021/12/09 14:03:35 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 14:03:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 14:03:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:03:40 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 14:03:45 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 14:03:50 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 14:03:55 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 14:03:55 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/09 14:03:55 No eventing-con process running
2021/12/09 14:03:55 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/09 14:03:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:04:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 14:04:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:04:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:04:19 Waiting for dst bucket item count to get to: 0 curr count: 9368
2021/12/09 14:04:24 Waiting for dst bucket item count to get to: 0 curr count: 4368
2021/12/09 14:04:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.03s)
=== RUN   TestN1QLThrowStatement
2021/12/09 14:04:29 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 14:04:29 App: TestN1QLThrowStatement got un-deployed
2021/12/09 14:04:29 No eventing-con process running
2021/12/09 14:04:29 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/09 14:04:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:04:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:04:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:04:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:04:42 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:04:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/09 14:04:48 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 14:04:53 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 14:04:58 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 14:05:03 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 14:05:03 App: TestN1QLThrowStatement got deployed
2021/12/09 14:05:03 Starting bucket ops to source bucket
2021/12/09 14:05:13 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/09 14:05:18 Waiting for dst bucket item count to get to: 10000 curr count: 259
2021/12/09 14:05:23 Waiting for dst bucket item count to get to: 10000 curr count: 846
2021/12/09 14:05:28 Waiting for dst bucket item count to get to: 10000 curr count: 2788
2021/12/09 14:05:33 Waiting for dst bucket item count to get to: 10000 curr count: 4824
2021/12/09 14:05:38 Waiting for dst bucket item count to get to: 10000 curr count: 6847
2021/12/09 14:05:43 Waiting for dst bucket item count to get to: 10000 curr count: 8860
2021/12/09 14:05:48 Waiting for dst bucket item count to get to: 10000 curr count: 9636
2021/12/09 14:05:48 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 14:05:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 14:05:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:05:53 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 14:05:58 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 14:06:03 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 14:06:08 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 14:06:08 App: TestN1QLThrowStatement got un-deployed
2021/12/09 14:06:08 No eventing-con process running
2021/12/09 14:06:08 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/09 14:06:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:06:20 Waiting for dst bucket item count to get to: 0 curr count: 2810
2021/12/09 14:06:25 Waiting for dst bucket item count to get to: 0 curr count: 2810
2021/12/09 14:06:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:06:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:06:37 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 14:06:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.03s)
=== RUN   TestN1QLNestedForLoop
2021/12/09 14:06:42 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 14:06:42 App: TestN1QLNestedForLoop got un-deployed
2021/12/09 14:06:42 No eventing-con process running
2021/12/09 14:06:42 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/09 14:06:44 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:06:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:06:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:06:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:06:55 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:06:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/09 14:07:02 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 14:07:07 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 14:07:12 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 14:07:17 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 14:07:17 App: TestN1QLNestedForLoop got deployed
2021/12/09 14:07:17 Starting bucket ops to source bucket
2021/12/09 14:07:26 Waiting for dst bucket item count to get to: 5000 curr count: 105
2021/12/09 14:07:31 Waiting for dst bucket item count to get to: 5000 curr count: 105
2021/12/09 14:07:36 Waiting for dst bucket item count to get to: 5000 curr count: 105
2021/12/09 14:07:41 Waiting for dst bucket item count to get to: 5000 curr count: 2034
2021/12/09 14:07:46 Waiting for dst bucket item count to get to: 5000 curr count: 2034
2021/12/09 14:07:51 Waiting for dst bucket item count to get to: 5000 curr count: 4410
2021/12/09 14:07:56 Waiting for dst bucket item count to get to: 5000 curr count: 4410
2021/12/09 14:07:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 14:07:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 14:07:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:08:01 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 14:08:06 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 14:08:11 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 14:08:16 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 14:08:16 App: TestN1QLNestedForLoop got un-deployed
2021/12/09 14:08:16 No eventing-con process running
2021/12/09 14:08:16 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/09 14:08:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:08:28 Waiting for dst bucket item count to get to: 0 curr count: 3802
2021/12/09 14:08:33 Waiting for dst bucket item count to get to: 0 curr count: 1302
2021/12/09 14:08:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:08:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:08:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 14:08:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.15s)
=== RUN   TestN1QLAutoClose
2021/12/09 14:08:50 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 14:08:50 App: TestN1QLAutoClose got un-deployed
2021/12/09 14:08:50 No eventing-con process running
2021/12/09 14:08:50 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/09 14:08:52 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:08:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:08:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:09:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:09:04 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:09:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/09 14:09:10 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 14:09:15 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 14:09:20 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 14:09:25 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 14:09:25 App: TestN1QLAutoClose got deployed
2021/12/09 14:09:25 Starting bucket ops to source bucket
2021/12/09 14:09:30 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/09 14:09:35 Waiting for dst bucket item count to get to: 500 curr count: 2
2021/12/09 14:09:40 Waiting for dst bucket item count to get to: 500 curr count: 5
2021/12/09 14:09:45 Waiting for dst bucket item count to get to: 500 curr count: 16
2021/12/09 14:09:50 Waiting for dst bucket item count to get to: 500 curr count: 29
2021/12/09 14:09:55 Waiting for dst bucket item count to get to: 500 curr count: 39
2021/12/09 14:10:00 Waiting for dst bucket item count to get to: 500 curr count: 51
2021/12/09 14:10:05 Waiting for dst bucket item count to get to: 500 curr count: 73
2021/12/09 14:10:10 Waiting for dst bucket item count to get to: 500 curr count: 73
2021/12/09 14:10:15 Waiting for dst bucket item count to get to: 500 curr count: 94
2021/12/09 14:10:20 Waiting for dst bucket item count to get to: 500 curr count: 94
2021/12/09 14:10:25 Waiting for dst bucket item count to get to: 500 curr count: 116
2021/12/09 14:10:30 Waiting for dst bucket item count to get to: 500 curr count: 116
2021/12/09 14:10:35 Waiting for dst bucket item count to get to: 500 curr count: 116
2021/12/09 14:10:40 Waiting for dst bucket item count to get to: 500 curr count: 137
2021/12/09 14:10:45 Waiting for dst bucket item count to get to: 500 curr count: 137
2021/12/09 14:10:50 Waiting for dst bucket item count to get to: 500 curr count: 159
2021/12/09 14:10:55 Waiting for dst bucket item count to get to: 500 curr count: 159
2021/12/09 14:11:00 Waiting for dst bucket item count to get to: 500 curr count: 181
2021/12/09 14:11:05 Waiting for dst bucket item count to get to: 500 curr count: 181
2021/12/09 14:11:10 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/09 14:11:15 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/09 14:11:20 Waiting for dst bucket item count to get to: 500 curr count: 224
2021/12/09 14:11:26 Waiting for dst bucket item count to get to: 500 curr count: 224
2021/12/09 14:11:31 Waiting for dst bucket item count to get to: 500 curr count: 246
2021/12/09 14:11:36 Waiting for dst bucket item count to get to: 500 curr count: 246
2021/12/09 14:11:41 Waiting for dst bucket item count to get to: 500 curr count: 266
2021/12/09 14:11:46 Waiting for dst bucket item count to get to: 500 curr count: 266
2021/12/09 14:11:51 Waiting for dst bucket item count to get to: 500 curr count: 288
2021/12/09 14:11:56 Waiting for dst bucket item count to get to: 500 curr count: 288
2021/12/09 14:12:01 Waiting for dst bucket item count to get to: 500 curr count: 310
2021/12/09 14:12:06 Waiting for dst bucket item count to get to: 500 curr count: 310
2021/12/09 14:12:11 Waiting for dst bucket item count to get to: 500 curr count: 333
2021/12/09 14:12:16 Waiting for dst bucket item count to get to: 500 curr count: 333
2021/12/09 14:12:21 Waiting for dst bucket item count to get to: 500 curr count: 354
2021/12/09 14:12:26 Waiting for dst bucket item count to get to: 500 curr count: 365
2021/12/09 14:12:31 Waiting for dst bucket item count to get to: 500 curr count: 374
2021/12/09 14:12:36 Waiting for dst bucket item count to get to: 500 curr count: 384
2021/12/09 14:12:41 Waiting for dst bucket item count to get to: 500 curr count: 396
2021/12/09 14:12:46 Waiting for dst bucket item count to get to: 500 curr count: 407
2021/12/09 14:12:51 Waiting for dst bucket item count to get to: 500 curr count: 417
2021/12/09 14:12:56 Waiting for dst bucket item count to get to: 500 curr count: 431
2021/12/09 14:13:01 Waiting for dst bucket item count to get to: 500 curr count: 439
2021/12/09 14:13:06 Waiting for dst bucket item count to get to: 500 curr count: 450
2021/12/09 14:13:11 Waiting for dst bucket item count to get to: 500 curr count: 459
2021/12/09 14:13:16 Waiting for dst bucket item count to get to: 500 curr count: 470
2021/12/09 14:13:21 Waiting for dst bucket item count to get to: 500 curr count: 483
2021/12/09 14:13:26 Waiting for dst bucket item count to get to: 500 curr count: 492
2021/12/09 14:13:31 Waiting for dst bucket item count to get to: 500 curr count: 499
2021/12/09 14:13:31 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/09 14:13:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 14:13:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:13:36 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 14:13:41 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 14:13:46 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 14:13:51 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 14:13:51 App: TestN1QLAutoClose got un-deployed
2021/12/09 14:13:51 No eventing-con process running
2021/12/09 14:13:51 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/09 14:13:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:14:03 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 14:14:08 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/09 14:14:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:14:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:14:20 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/09 14:14:25 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/09 14:14:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.64s)
=== RUN   TestN1QLPosParams
2021/12/09 14:14:30 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 14:14:30 App: TestN1QLPosParams got un-deployed
2021/12/09 14:14:30 No eventing-con process running
2021/12/09 14:14:30 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/09 14:14:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:14:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:14:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:14:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:14:43 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:14:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/09 14:14:49 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 14:14:54 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 14:14:59 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 14:15:04 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 14:15:04 App: TestN1QLPosParams got deployed
2021/12/09 14:15:04 Starting bucket ops to source bucket
2021/12/09 14:15:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 14:15:19 Waiting for dst bucket item count to get to: 5000 curr count: 462
2021/12/09 14:15:24 Waiting for dst bucket item count to get to: 5000 curr count: 1417
2021/12/09 14:15:29 Waiting for dst bucket item count to get to: 5000 curr count: 3455
2021/12/09 14:15:29 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 14:15:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 14:15:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:15:34 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 14:15:39 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 14:15:44 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 14:15:49 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 14:15:49 App: TestN1QLPosParams got un-deployed
2021/12/09 14:15:49 No eventing-con process running
2021/12/09 14:15:49 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/09 14:15:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:16:01 Waiting for dst bucket item count to get to: 0 curr count: 4244
2021/12/09 14:16:06 Waiting for dst bucket item count to get to: 0 curr count: 4244
2021/12/09 14:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:16:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:16:17 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 14:16:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (112.83s)
=== RUN   TestN1QLExhaustConnPool
2021/12/09 14:16:22 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 14:16:22 App: TestN1QLExhaustConnPool got un-deployed
2021/12/09 14:16:22 No eventing-con process running
2021/12/09 14:16:22 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/09 14:16:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:16:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:16:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:16:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:16:36 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:16:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/09 14:16:42 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 14:16:47 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 14:16:52 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 14:16:57 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 14:16:57 App: TestN1QLExhaustConnPool got deployed
2021/12/09 14:16:57 Starting bucket ops to source bucket
2021/12/09 14:17:02 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 14:17:07 Waiting for dst bucket item count to get to: 100 curr count: 34
2021/12/09 14:17:12 Waiting for dst bucket item count to get to: 100 curr count: 34
2021/12/09 14:17:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 14:17:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 14:17:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:17:18 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 14:17:23 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 14:17:28 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 14:17:33 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 14:17:33 App: TestN1QLExhaustConnPool got un-deployed
2021/12/09 14:17:33 No eventing-con process running
2021/12/09 14:17:33 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/09 14:17:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:17:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 14:17:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:17:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:17:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 14:17:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (98.77s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/09 14:18:01 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 14:18:01 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/09 14:18:01 No eventing-con process running
2021/12/09 14:18:01 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/09 14:18:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:18:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:18:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:18:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:18:15 Starting bucket ops to source bucket
2021/12/09 14:18:32 Starting bucket ops to source bucket
2021/12/09 14:18:49 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 11266
2021/12/09 14:18:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/09 14:18:55 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 14:19:00 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 14:19:05 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 14:19:10 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 14:19:10 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/09 14:19:10 Starting bucket ops to source bucket
2021/12/09 14:19:15 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 14:19:20 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 14:19:25 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 14:19:30 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 14:19:35 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 14:19:40 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 14:19:45 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 14:19:50 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 14:19:55 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 14:20:00 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 14:20:05 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 14:20:05 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/09 14:20:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 14:20:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:20:11 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 14:20:16 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 14:20:21 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 14:20:26 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 14:20:26 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/09 14:20:26 No eventing-con process running
2021/12/09 14:20:26 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/09 14:20:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:20:37 Waiting for dst bucket item count to get to: 0 curr count: 15000
2021/12/09 14:20:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:20:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:20:49 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/09 14:20:54 Waiting for dst bucket item count to get to: 0 curr count: 15001
2021/12/09 14:20:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (178.07s)
=== RUN   TestN1QLQueryConsistency
2021/12/09 14:20:59 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 14:20:59 App: n1qlquery_consistency got un-deployed
2021/12/09 14:20:59 No eventing-con process running
2021/12/09 14:20:59 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/09 14:21:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:21:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:21:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:21:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:21:13 Starting bucket ops to source bucket
2021/12/09 14:21:16 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:21:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/09 14:21:22 Waiting for app: n1qlquery_consistency to get deployed
2021/12/09 14:21:27 Waiting for app: n1qlquery_consistency to get deployed
2021/12/09 14:21:32 Waiting for app: n1qlquery_consistency to get deployed
2021/12/09 14:21:37 Waiting for app: n1qlquery_consistency to get deployed
2021/12/09 14:21:37 App: n1qlquery_consistency got deployed
2021/12/09 14:21:42 Waiting for dst bucket item count to get to: 5000 curr count: 98
2021/12/09 14:21:47 Waiting for dst bucket item count to get to: 5000 curr count: 2598
2021/12/09 14:21:47 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 14:21:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 14:21:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:21:52 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 14:21:57 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 14:22:02 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 14:22:07 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 14:22:07 App: n1qlquery_consistency got un-deployed
2021/12/09 14:22:07 No eventing-con process running
2021/12/09 14:22:07 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/09 14:22:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:22:19 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 14:22:24 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 14:22:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:22:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 14:22:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 14:22:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.59s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1556.476s

N1ql tests took 0 hours, 26 minutes and 1 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/09 14:24:11 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:40558->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:40558->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/09 14:24:29 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/09 14:24:34 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 14:24:34 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/09 14:24:34 No eventing-con process running
2021/12/09 14:24:34 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/09 14:24:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:24:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:24:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:24:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:24:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:24:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:24:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:24:53 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:24:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 14:24:59 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 14:25:04 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 14:25:09 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 14:25:14 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 14:25:14 App: TestDuplicateWithEventingRebIn got deployed
2021/12/09 14:25:14 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 14:25:14 Starting bucket ops to source bucket
2021/12/09 14:25:21 Removing node(s): [] from the cluster
2021/12/09 14:25:27 Rebalance progress: 0
2021/12/09 14:25:32 Rebalance progress: 0
2021/12/09 14:25:37 Rebalance progress: 25
2021/12/09 14:25:42 Rebalance progress: 25
2021/12/09 14:25:47 Rebalance progress: 25
2021/12/09 14:25:52 Rebalance progress: 25
2021/12/09 14:25:57 Rebalance progress: 25
2021/12/09 14:26:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:26:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:26:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:26:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 14:26:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:26:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 14:26:07 Rebalance progress: 25
2021/12/09 14:26:12 Rebalance progress: 25
2021/12/09 14:26:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:26:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:26:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:26:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 14:26:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:26:17 Waiting for src bucket item count to get to: 200000 curr count: 90027
2021/12/09 14:26:22 Waiting for src bucket item count to get to: 200000 curr count: 90027
2021/12/09 14:26:27 Waiting for src bucket item count to get to: 200000 curr count: 106781
2021/12/09 14:26:32 Waiting for src bucket item count to get to: 200000 curr count: 106781
2021/12/09 14:26:37 Waiting for src bucket item count to get to: 200000 curr count: 123584
2021/12/09 14:26:42 Waiting for src bucket item count to get to: 200000 curr count: 140428
2021/12/09 14:26:47 Waiting for src bucket item count to get to: 200000 curr count: 140428
2021/12/09 14:26:52 Waiting for src bucket item count to get to: 200000 curr count: 156817
2021/12/09 14:26:57 Waiting for src bucket item count to get to: 200000 curr count: 156817
2021/12/09 14:27:02 Waiting for src bucket item count to get to: 200000 curr count: 173787
2021/12/09 14:27:07 Waiting for src bucket item count to get to: 200000 curr count: 173787
2021/12/09 14:27:12 Waiting for src bucket item count to get to: 200000 curr count: 190378
2021/12/09 14:27:17 Waiting for src bucket item count to get to: 200000 curr count: 190378
2021/12/09 14:27:22 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 14:27:22 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 14:27:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:27:27 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 14:27:32 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 14:27:37 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 14:27:42 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 14:27:42 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/09 14:27:42 No eventing-con process running
2021/12/09 14:27:42 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/09 14:27:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:27:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:27:54 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 14:27:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 14:27:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:28:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:28:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:28:10 Waiting for dst bucket item count to get to: 0 curr count: 72177
2021/12/09 14:28:15 Waiting for dst bucket item count to get to: 0 curr count: 72177
2021/12/09 14:28:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (235.87s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/09 14:28:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 14:28:21 Removing node(s): [] from the cluster
2021/12/09 14:28:26 Rebalance progress: 100
2021/12/09 14:28:31 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 14:28:31 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/09 14:28:31 No eventing-con process running
2021/12/09 14:28:31 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/09 14:28:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:28:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:28:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:28:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:28:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:28:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:28:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:28:49 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:28:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 14:28:55 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 14:29:00 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 14:29:05 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 14:29:10 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 14:29:10 App: TestDuplicateWithEventingRebOut got deployed
2021/12/09 14:29:10 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 14:29:10 Starting bucket ops to source bucket
2021/12/09 14:29:15 Rebalance progress: 25
2021/12/09 14:29:20 Rebalance progress: 25
2021/12/09 14:29:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:29:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:29:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:29:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 14:29:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:29:25 Waiting for src bucket item count to get to: 200000 curr count: 14651
2021/12/09 14:29:30 Waiting for src bucket item count to get to: 200000 curr count: 31245
2021/12/09 14:29:35 Waiting for src bucket item count to get to: 200000 curr count: 31245
2021/12/09 14:29:40 Waiting for src bucket item count to get to: 200000 curr count: 48127
2021/12/09 14:29:45 Waiting for src bucket item count to get to: 200000 curr count: 48127
2021/12/09 14:29:50 Waiting for src bucket item count to get to: 200000 curr count: 65207
2021/12/09 14:29:55 Waiting for src bucket item count to get to: 200000 curr count: 65207
2021/12/09 14:30:00 Waiting for src bucket item count to get to: 200000 curr count: 65207
2021/12/09 14:30:05 Waiting for src bucket item count to get to: 200000 curr count: 81527
2021/12/09 14:30:10 Waiting for src bucket item count to get to: 200000 curr count: 81527
2021/12/09 14:30:15 Waiting for src bucket item count to get to: 200000 curr count: 98089
2021/12/09 14:30:20 Waiting for src bucket item count to get to: 200000 curr count: 98089
2021/12/09 14:30:25 Waiting for src bucket item count to get to: 200000 curr count: 114899
2021/12/09 14:30:30 Waiting for src bucket item count to get to: 200000 curr count: 114899
2021/12/09 14:30:35 Waiting for src bucket item count to get to: 200000 curr count: 132123
2021/12/09 14:30:40 Waiting for src bucket item count to get to: 200000 curr count: 132123
2021/12/09 14:30:45 Waiting for src bucket item count to get to: 200000 curr count: 148879
2021/12/09 14:30:50 Waiting for src bucket item count to get to: 200000 curr count: 148879
2021/12/09 14:30:56 Waiting for src bucket item count to get to: 200000 curr count: 165938
2021/12/09 14:31:01 Waiting for src bucket item count to get to: 200000 curr count: 165938
2021/12/09 14:31:06 Waiting for src bucket item count to get to: 200000 curr count: 182323
2021/12/09 14:31:11 Waiting for src bucket item count to get to: 200000 curr count: 182323
2021/12/09 14:31:16 Waiting for src bucket item count to get to: 200000 curr count: 199130
2021/12/09 14:31:21 Waiting for src bucket item count to get to: 200000 curr count: 199130
2021/12/09 14:31:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 14:31:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 14:31:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:31:31 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 14:31:36 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 14:31:41 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 14:31:46 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 14:31:46 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/09 14:31:46 No eventing-con process running
2021/12/09 14:31:46 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/09 14:31:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:31:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:31:57 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 14:31:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:31:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:32:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:32:09 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 14:32:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (233.79s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/09 14:32:19 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 14:32:19 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/09 14:32:19 No eventing-con process running
2021/12/09 14:32:19 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/09 14:32:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:32:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:32:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:32:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:32:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:32:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:32:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:32:38 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:32:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 14:32:44 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 14:32:49 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 14:32:54 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 14:32:59 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 14:32:59 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/09 14:32:59 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 14:32:59 Starting bucket ops to source bucket
2021/12/09 14:32:59 Removing node(s): [] from the cluster
2021/12/09 14:33:04 Rebalance progress: 25
2021/12/09 14:33:09 Rebalance progress: 25
2021/12/09 14:33:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:33:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:33:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:33:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 14:33:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:33:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 14:33:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 14:33:27 Rebalance progress: 16.66666666666667
2021/12/09 14:33:32 Rebalance progress: 16.66666666666667
2021/12/09 14:33:37 Rebalance progress: 16.66666666666667
2021/12/09 14:33:42 Rebalance progress: 73.1359649122807
2021/12/09 14:33:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:33:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:33:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:33:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:33:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 14:33:52 Rebalance progress: 16.66666666666667
2021/12/09 14:33:57 Rebalance progress: 16.66666666666667
2021/12/09 14:34:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:34:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:34:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:34:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:34:02 Waiting for src bucket item count to get to: 200000 curr count: 77374
2021/12/09 14:34:07 Waiting for src bucket item count to get to: 200000 curr count: 93927
2021/12/09 14:34:12 Waiting for src bucket item count to get to: 200000 curr count: 93927
2021/12/09 14:34:17 Waiting for src bucket item count to get to: 200000 curr count: 110428
2021/12/09 14:34:22 Waiting for src bucket item count to get to: 200000 curr count: 110428
2021/12/09 14:34:27 Waiting for src bucket item count to get to: 200000 curr count: 127254
2021/12/09 14:34:32 Waiting for src bucket item count to get to: 200000 curr count: 127254
2021/12/09 14:34:37 Waiting for src bucket item count to get to: 200000 curr count: 143780
2021/12/09 14:34:42 Waiting for src bucket item count to get to: 200000 curr count: 160024
2021/12/09 14:34:47 Waiting for src bucket item count to get to: 200000 curr count: 160024
2021/12/09 14:34:52 Waiting for src bucket item count to get to: 200000 curr count: 175939
2021/12/09 14:34:57 Waiting for src bucket item count to get to: 200000 curr count: 175939
2021/12/09 14:35:02 Waiting for src bucket item count to get to: 200000 curr count: 192519
2021/12/09 14:35:07 Waiting for src bucket item count to get to: 200000 curr count: 192519
2021/12/09 14:35:12 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 14:35:12 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 14:35:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:35:17 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 14:35:22 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 14:35:27 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 14:35:32 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 14:35:32 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/09 14:35:32 No eventing-con process running
2021/12/09 14:35:32 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/09 14:35:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:35:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:35:44 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 14:35:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 14:35:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:35:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:35:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:36:00 Waiting for dst bucket item count to get to: 0 curr count: 78075
2021/12/09 14:36:05 Waiting for dst bucket item count to get to: 0 curr count: 78075
2021/12/09 14:36:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (236.22s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/09 14:36:15 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 14:36:15 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/09 14:36:15 No eventing-con process running
2021/12/09 14:36:15 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/09 14:36:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:36:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:36:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:36:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:36:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:36:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:36:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:36:34 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:36:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 14:36:40 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 14:36:45 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 14:36:50 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 14:36:55 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 14:36:55 App: TestDuplicateWithKVRebIn got deployed
2021/12/09 14:36:55 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 14:36:55 Starting bucket ops to source bucket
2021/12/09 14:36:55 Removing node(s): [] from the cluster
2021/12/09 14:37:00 Rebalance progress: 16.66666666666667
2021/12/09 14:37:05 Rebalance progress: 16.66666666666667
2021/12/09 14:37:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:37:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:37:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:37:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:37:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 14:37:15 Rebalance progress: 16.66666666666667
2021/12/09 14:37:20 Rebalance progress: 16.66666666666667
2021/12/09 14:37:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:37:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:37:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:37:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:37:26 Waiting for src bucket item count to get to: 200000 curr count: 38940
2021/12/09 14:37:31 Waiting for src bucket item count to get to: 200000 curr count: 55376
2021/12/09 14:37:36 Waiting for src bucket item count to get to: 200000 curr count: 55376
2021/12/09 14:37:41 Waiting for src bucket item count to get to: 200000 curr count: 71871
2021/12/09 14:37:46 Waiting for src bucket item count to get to: 200000 curr count: 71871
2021/12/09 14:37:51 Waiting for src bucket item count to get to: 200000 curr count: 87619
2021/12/09 14:37:56 Waiting for src bucket item count to get to: 200000 curr count: 87619
2021/12/09 14:38:01 Waiting for src bucket item count to get to: 200000 curr count: 104288
2021/12/09 14:38:06 Waiting for src bucket item count to get to: 200000 curr count: 104288
2021/12/09 14:38:11 Waiting for src bucket item count to get to: 200000 curr count: 121048
2021/12/09 14:38:16 Waiting for src bucket item count to get to: 200000 curr count: 121048
2021/12/09 14:38:21 Waiting for src bucket item count to get to: 200000 curr count: 121048
2021/12/09 14:38:26 Waiting for src bucket item count to get to: 200000 curr count: 137432
2021/12/09 14:38:31 Waiting for src bucket item count to get to: 200000 curr count: 137432
2021/12/09 14:38:36 Waiting for src bucket item count to get to: 200000 curr count: 153815
2021/12/09 14:38:41 Waiting for src bucket item count to get to: 200000 curr count: 153815
2021/12/09 14:38:46 Waiting for src bucket item count to get to: 200000 curr count: 170258
2021/12/09 14:38:51 Waiting for src bucket item count to get to: 200000 curr count: 170258
2021/12/09 14:38:56 Waiting for src bucket item count to get to: 200000 curr count: 185921
2021/12/09 14:39:01 Waiting for src bucket item count to get to: 200000 curr count: 185921
2021/12/09 14:39:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 14:39:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 14:39:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:39:11 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 14:39:16 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 14:39:21 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 14:39:26 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 14:39:26 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/09 14:39:26 No eventing-con process running
2021/12/09 14:39:26 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/09 14:39:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:39:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:39:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 14:39:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:39:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:39:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:39:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (219.05s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/09 14:39:49 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 14:39:49 Removing node(s): [] from the cluster
2021/12/09 14:39:55 Rebalance progress: 100
2021/12/09 14:40:00 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 14:40:00 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/09 14:40:00 No eventing-con process running
2021/12/09 14:40:00 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/09 14:40:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:40:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:40:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:40:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:40:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:40:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:40:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:40:18 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:40:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 14:40:24 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 14:40:29 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 14:40:34 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 14:40:39 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 14:40:39 App: TestDuplicateWithKVRebOut got deployed
2021/12/09 14:40:39 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 14:40:39 Starting bucket ops to source bucket
2021/12/09 14:40:44 Rebalance progress: 16.66666666666667
2021/12/09 14:40:49 Rebalance progress: 16.66666666666667
2021/12/09 14:40:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:40:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:40:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:40:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:40:54 Waiting for src bucket item count to get to: 200000 curr count: 15941
2021/12/09 14:40:59 Waiting for src bucket item count to get to: 200000 curr count: 15941
2021/12/09 14:41:04 Waiting for src bucket item count to get to: 200000 curr count: 32547
2021/12/09 14:41:09 Waiting for src bucket item count to get to: 200000 curr count: 32547
2021/12/09 14:41:14 Waiting for src bucket item count to get to: 200000 curr count: 48890
2021/12/09 14:41:19 Waiting for src bucket item count to get to: 200000 curr count: 48890
2021/12/09 14:41:24 Waiting for src bucket item count to get to: 200000 curr count: 48890
2021/12/09 14:41:29 Waiting for src bucket item count to get to: 200000 curr count: 65357
2021/12/09 14:41:34 Waiting for src bucket item count to get to: 200000 curr count: 65357
2021/12/09 14:41:39 Waiting for src bucket item count to get to: 200000 curr count: 81101
2021/12/09 14:41:44 Waiting for src bucket item count to get to: 200000 curr count: 97164
2021/12/09 14:41:49 Waiting for src bucket item count to get to: 200000 curr count: 97164
2021/12/09 14:41:54 Waiting for src bucket item count to get to: 200000 curr count: 113552
2021/12/09 14:41:59 Waiting for src bucket item count to get to: 200000 curr count: 113552
2021/12/09 14:42:04 Waiting for src bucket item count to get to: 200000 curr count: 130070
2021/12/09 14:42:09 Waiting for src bucket item count to get to: 200000 curr count: 130070
2021/12/09 14:42:14 Waiting for src bucket item count to get to: 200000 curr count: 146565
2021/12/09 14:42:19 Waiting for src bucket item count to get to: 200000 curr count: 146565
2021/12/09 14:42:24 Waiting for src bucket item count to get to: 200000 curr count: 162923
2021/12/09 14:42:29 Waiting for src bucket item count to get to: 200000 curr count: 162923
2021/12/09 14:42:34 Waiting for src bucket item count to get to: 200000 curr count: 178684
2021/12/09 14:42:39 Waiting for src bucket item count to get to: 200000 curr count: 178684
2021/12/09 14:42:44 Waiting for src bucket item count to get to: 200000 curr count: 195153
2021/12/09 14:42:49 Waiting for src bucket item count to get to: 200000 curr count: 195153
2021/12/09 14:42:54 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 14:42:54 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 14:42:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:43:00 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 14:43:05 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 14:43:10 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 14:43:15 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 14:43:15 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/09 14:43:15 No eventing-con process running
2021/12/09 14:43:15 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/09 14:43:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:43:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:43:26 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 14:43:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:43:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:43:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:43:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 14:43:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (233.79s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/09 14:43:48 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 14:43:48 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/09 14:43:48 No eventing-con process running
2021/12/09 14:43:48 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/09 14:43:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:43:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:43:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:43:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:44:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:44:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:44:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:44:07 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:44:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 14:44:13 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 14:44:18 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 14:44:23 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 14:44:28 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 14:44:28 App: TestDuplicateWithKVSwapReb got deployed
2021/12/09 14:44:28 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 14:44:28 Starting bucket ops to source bucket
2021/12/09 14:44:28 Removing node(s): [] from the cluster
2021/12/09 14:44:33 Rebalance progress: 16.66666666666667
2021/12/09 14:44:38 Rebalance progress: 16.66666666666667
2021/12/09 14:44:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:44:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:44:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:44:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:44:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 14:44:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 14:44:49 Rebalance progress: 16.66666666666667
2021/12/09 14:44:54 Rebalance progress: 16.66666666666667
2021/12/09 14:44:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:44:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:44:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:44:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:44:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 14:45:04 Rebalance progress: 16.66666666666667
2021/12/09 14:45:09 Rebalance progress: 16.66666666666667
2021/12/09 14:45:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:45:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:45:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:45:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:45:14 Waiting for src bucket item count to get to: 200000 curr count: 65953
2021/12/09 14:45:19 Waiting for src bucket item count to get to: 200000 curr count: 65953
2021/12/09 14:45:24 Waiting for src bucket item count to get to: 200000 curr count: 81037
2021/12/09 14:45:29 Waiting for src bucket item count to get to: 200000 curr count: 81037
2021/12/09 14:45:34 Waiting for src bucket item count to get to: 200000 curr count: 97605
2021/12/09 14:45:39 Waiting for src bucket item count to get to: 200000 curr count: 97605
2021/12/09 14:45:44 Waiting for src bucket item count to get to: 200000 curr count: 114299
2021/12/09 14:45:49 Waiting for src bucket item count to get to: 200000 curr count: 114299
2021/12/09 14:45:54 Waiting for src bucket item count to get to: 200000 curr count: 131101
2021/12/09 14:45:59 Waiting for src bucket item count to get to: 200000 curr count: 131101
2021/12/09 14:46:04 Waiting for src bucket item count to get to: 200000 curr count: 147841
2021/12/09 14:46:09 Waiting for src bucket item count to get to: 200000 curr count: 147841
2021/12/09 14:46:14 Waiting for src bucket item count to get to: 200000 curr count: 164333
2021/12/09 14:46:19 Waiting for src bucket item count to get to: 200000 curr count: 164333
2021/12/09 14:46:24 Waiting for src bucket item count to get to: 200000 curr count: 164333
2021/12/09 14:46:29 Waiting for src bucket item count to get to: 200000 curr count: 180238
2021/12/09 14:46:34 Waiting for src bucket item count to get to: 200000 curr count: 180238
2021/12/09 14:46:39 Waiting for src bucket item count to get to: 200000 curr count: 196737
2021/12/09 14:46:44 Waiting for src bucket item count to get to: 200000 curr count: 196737
2021/12/09 14:46:49 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 14:46:49 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 14:46:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:46:54 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 14:46:59 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 14:47:04 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 14:47:09 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 14:47:09 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/09 14:47:09 No eventing-con process running
2021/12/09 14:47:09 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/09 14:47:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:47:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:47:21 Waiting for dst bucket item count to get to: 0 curr count: 171312
2021/12/09 14:47:26 Waiting for dst bucket item count to get to: 0 curr count: 171312
2021/12/09 14:47:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:47:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:47:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:47:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 14:47:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (239.43s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/09 14:47:48 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 14:47:48 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 14:47:48 No eventing-con process running
2021/12/09 14:47:48 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/09 14:47:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:47:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:47:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:47:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:48:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:48:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:48:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:48:06 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:48:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/09 14:48:12 Starting bucket ops to source bucket
2021/12/09 14:48:17 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 14:48:22 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 14:48:27 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 14:48:27 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:48:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:48:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:48:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:48:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 14:48:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 14:48:32 Rebalance progress: 16.66666666666667
2021/12/09 14:48:37 Rebalance progress: 16.66666666666667
2021/12/09 14:48:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:48:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:48:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:48:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:48:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 14:48:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 14:48:48 Rebalance progress: 16.66666666666667
2021/12/09 14:48:53 Rebalance progress: 16.66666666666667
2021/12/09 14:48:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:48:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:48:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:48:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 14:48:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 14:49:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 14:49:08 Rebalance progress: 12.5
2021/12/09 14:49:13 Rebalance progress: 12.5
2021/12/09 14:49:18 Rebalance progress: 12.5
2021/12/09 14:49:23 Rebalance progress: 32.63888888888889
2021/12/09 14:49:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:49:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:49:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:49:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 14:49:33 Rebalance progress: 12.5
2021/12/09 14:49:38 Rebalance progress: 12.5
2021/12/09 14:49:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:49:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 14:49:49 Rebalance progress: 12.5
2021/12/09 14:49:54 Rebalance progress: 12.5
2021/12/09 14:49:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:49:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:49:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:49:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 14:50:04 Rebalance progress: 12.5
2021/12/09 14:50:09 Rebalance progress: 12.5
2021/12/09 14:50:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:50:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:50:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:50:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:50:19 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 14:50:24 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 14:50:29 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 14:50:34 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 14:50:34 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 14:50:34 eventingSetup.go:46: No eventing-con process running
2021/12/09 14:50:34 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/09 14:50:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:50:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:50:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12150
2021/12/09 14:50:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:50:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:50:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:50:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12150
2021/12/09 14:50:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (199.86s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/09 14:51:07 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 14:51:07 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/09 14:51:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 14:51:07 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/09 14:51:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:51:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:51:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:51:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:51:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:51:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:51:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:51:26 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:51:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/09 14:51:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 14:51:37 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/09 14:51:42 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/09 14:51:47 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/09 14:51:47 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:51:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:51:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:51:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 14:51:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 14:51:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 14:51:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 14:51:53 Rebalance progress: 12.5
2021/12/09 14:51:58 Rebalance progress: 12.5
2021/12/09 14:52:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:52:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:52:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:52:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 14:52:08 Rebalance progress: 12.5
2021/12/09 14:52:13 Rebalance progress: 12.5
2021/12/09 14:52:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:52:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:52:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:52:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:52:23 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 14:52:28 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 14:52:33 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 14:52:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 14:52:38 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/09 14:52:38 eventingSetup.go:46: No eventing-con process running
2021/12/09 14:52:38 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/09 14:52:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:52:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:52:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 189
2021/12/09 14:52:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 189
2021/12/09 14:52:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:52:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:53:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:53:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4576
2021/12/09 14:53:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.25s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/09 14:53:17 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 14:53:17 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 14:53:17 eventingSetup.go:46: No eventing-con process running
2021/12/09 14:53:17 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/09 14:53:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:53:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:53:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:53:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:53:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:53:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:53:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:53:35 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:53:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 14:53:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 14:53:46 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/09 14:53:51 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/09 14:53:56 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/09 14:53:56 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:53:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:53:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:53:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 14:53:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 14:54:02 Rebalance progress: 12.5
2021/12/09 14:54:07 Rebalance progress: 12.5
2021/12/09 14:54:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:54:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:54:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:54:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 14:54:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 14:54:17 Rebalance progress: 12.5
2021/12/09 14:54:22 Rebalance progress: 12.5
2021/12/09 14:54:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:54:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 14:54:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 14:54:33 Rebalance progress: 12.5
2021/12/09 14:54:38 Rebalance progress: 12.5
2021/12/09 14:54:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:54:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:54:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:54:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 14:54:48 Rebalance progress: 12.5
2021/12/09 14:54:53 Rebalance progress: 12.5
2021/12/09 14:54:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:54:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:54:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:54:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 14:55:03 Rebalance progress: 12.5
2021/12/09 14:55:08 Rebalance progress: 12.5
2021/12/09 14:55:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:55:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:55:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:55:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 14:55:18 Rebalance progress: 12.5
2021/12/09 14:55:23 Rebalance progress: 12.5
2021/12/09 14:55:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:55:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:55:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:55:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:55:33 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 14:55:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 14:55:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 14:55:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 14:55:48 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 14:55:48 eventingSetup.go:46: No eventing-con process running
2021/12/09 14:55:48 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/09 14:55:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:55:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:56:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2992
2021/12/09 14:56:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2992
2021/12/09 14:56:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:56:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:56:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:56:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10676
2021/12/09 14:56:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.28s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/09 14:56:27 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 14:56:27 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/09 14:56:27 eventingSetup.go:46: No eventing-con process running
2021/12/09 14:56:27 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/09 14:56:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:56:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:56:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:56:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:56:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:56:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:56:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:56:40 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:56:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 14:56:47 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 14:56:52 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/09 14:56:57 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/09 14:57:02 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/09 14:57:02 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:57:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:57:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:57:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 14:57:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 14:57:07 Rebalance progress: 12.5
2021/12/09 14:57:12 Rebalance progress: 12.5
2021/12/09 14:57:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:57:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:57:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:57:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 14:57:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 14:57:23 Rebalance progress: 12.5
2021/12/09 14:57:28 Rebalance progress: 12.5
2021/12/09 14:57:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:57:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:57:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:57:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 14:57:38 Rebalance progress: 12.5
2021/12/09 14:57:43 Rebalance progress: 12.5
2021/12/09 14:57:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:57:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:57:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:57:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:57:53 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 14:57:58 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 14:58:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 14:58:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 14:58:08 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/09 14:58:08 eventingSetup.go:46: No eventing-con process running
2021/12/09 14:58:08 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/09 14:58:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:58:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:58:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:58:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:58:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:58:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6094
2021/12/09 14:58:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (129.46s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/09 14:58:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 14:58:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 14:58:37 Rebalance progress: 100
2021/12/09 14:58:42 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 14:58:42 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/09 14:58:42 No eventing-con process running
2021/12/09 14:58:42 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/09 14:58:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:58:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:58:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 14:58:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:58:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:58:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:58:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 14:58:56 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 14:58:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/09 14:59:02 Starting bucket ops to source bucket
2021/12/09 14:59:07 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/09 14:59:12 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/09 14:59:17 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/09 14:59:17 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:59:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:59:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:59:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 14:59:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 14:59:22 Rebalance progress: 12.5
2021/12/09 14:59:27 Rebalance progress: 12.5
2021/12/09 14:59:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:59:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:59:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:59:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/09 14:59:32 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/09 14:59:37 Rebalance progress: 12.5
2021/12/09 14:59:42 Rebalance progress: 12.5
2021/12/09 14:59:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:59:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 14:59:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 14:59:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 14:59:52 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 14:59:57 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 15:00:02 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 15:00:07 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 15:00:07 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/09 15:00:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:00:08 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/09 15:00:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:00:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:00:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4563
2021/12/09 15:00:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:00:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:00:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:00:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4563
2021/12/09 15:00:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4563
2021/12/09 15:00:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.60s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/09 15:00:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:00:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:00:47 Rebalance progress: 100
2021/12/09 15:00:52 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 15:00:52 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/09 15:00:52 No eventing-con process running
2021/12/09 15:00:52 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/09 15:00:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:00:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:00:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:01:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:01:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:01:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:01:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:01:05 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:01:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/09 15:01:11 Starting bucket ops to source bucket
2021/12/09 15:01:16 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/09 15:01:21 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/09 15:01:26 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/09 15:01:26 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:01:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:01:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:01:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 15:01:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:01:32 Rebalance progress: 12.5
2021/12/09 15:01:37 Rebalance progress: 12.5
2021/12/09 15:01:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:01:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:01:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:01:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/09 15:01:42 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/09 15:01:47 Rebalance progress: 12.5
2021/12/09 15:01:52 Rebalance progress: 12.5
2021/12/09 15:01:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:01:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:01:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:01:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:01:57 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/09 15:02:02 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 15:02:07 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 15:02:12 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 15:02:17 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 15:02:17 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/09 15:02:17 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:02:17 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/09 15:02:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:02:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:02:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 38195
2021/12/09 15:02:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:02:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:02:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:02:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 19098
2021/12/09 15:02:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 19098
2021/12/09 15:02:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.41s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/09 15:02:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 15:02:55 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/09 15:02:55 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:02:55 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/09 15:02:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:03:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:03:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:03:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:03:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:03:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:03:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:03:14 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:03:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:03:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:03:15 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:03:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 15:03:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 15:03:26 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/09 15:03:31 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/09 15:03:36 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/09 15:03:36 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/09 15:03:41 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/09 15:03:46 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/09 15:03:51 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/09 15:03:51 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:03:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:03:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:03:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:03:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:03:57 Rebalance progress: 12.5
2021/12/09 15:04:02 Rebalance progress: 12.5
2021/12/09 15:04:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:04:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:04:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:04:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 15:04:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:04:12 Rebalance progress: 12.5
2021/12/09 15:04:17 Rebalance progress: 12.5
2021/12/09 15:04:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:04:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:04:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:04:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 15:04:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:04:28 Rebalance progress: 12.5
2021/12/09 15:04:33 Rebalance progress: 12.5
2021/12/09 15:04:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:04:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:04:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:04:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 15:04:43 Rebalance progress: 12.5
2021/12/09 15:04:48 Rebalance progress: 12.5
2021/12/09 15:04:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:04:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:04:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:04:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 15:04:58 Rebalance progress: 12.5
2021/12/09 15:05:03 Rebalance progress: 12.5
2021/12/09 15:05:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:05:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:05:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:05:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 15:05:13 Rebalance progress: 12.5
2021/12/09 15:05:18 Rebalance progress: 12.5
2021/12/09 15:05:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:05:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:05:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:05:23 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:05:28 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 15:05:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 15:05:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 15:05:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 15:05:43 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/09 15:05:43 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:05:43 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/09 15:05:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:05:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:05:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12186
2021/12/09 15:05:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:05:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:06:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:06:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12186
2021/12/09 15:06:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:06:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:06:14 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 15:06:19 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 15:06:24 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 15:06:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 15:06:29 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/09 15:06:29 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:06:29 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/09 15:06:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:06:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:06:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:06:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:06:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:06:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (236.80s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/09 15:06:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 15:06:52 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/09 15:06:52 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:06:52 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/09 15:06:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:06:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:06:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:07:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:07:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:07:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:07:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:07:11 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:07:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:07:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:07:12 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:07:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 15:07:18 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 15:07:23 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/09 15:07:28 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/09 15:07:33 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/09 15:07:33 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/09 15:07:38 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/09 15:07:43 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/09 15:07:48 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/09 15:07:48 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:07:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:07:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:07:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:07:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 15:07:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 15:07:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:07:55 Rebalance progress: 12.5
2021/12/09 15:08:00 Rebalance progress: 12.5
2021/12/09 15:08:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:08:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:08:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:08:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 15:08:10 Rebalance progress: 12.5
2021/12/09 15:08:15 Rebalance progress: 12.5
2021/12/09 15:08:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:08:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:08:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:08:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:08:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 15:08:30 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 15:08:35 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 15:08:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 15:08:40 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/09 15:08:40 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:08:40 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/09 15:08:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:08:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:08:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6170
2021/12/09 15:08:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6170
2021/12/09 15:08:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:09:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:09:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:09:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2399
2021/12/09 15:09:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2399
2021/12/09 15:09:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:09:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:09:20 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 15:09:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 15:09:30 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 15:09:35 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 15:09:35 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/09 15:09:35 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:09:36 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/09 15:09:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:09:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:09:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:09:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:09:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:09:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (186.82s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/09 15:09:59 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 15:09:59 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/09 15:09:59 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:09:59 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/09 15:10:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:10:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:10:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:10:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:10:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:10:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:10:18 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 15:10:18 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/09 15:10:18 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:10:18 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/09 15:10:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:10:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:10:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:10:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:10:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:10:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:10:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:10:31 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:10:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:10:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:10:32 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:10:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/09 15:10:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 15:10:43 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/09 15:10:48 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/09 15:10:53 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/09 15:10:53 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/09 15:10:58 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/09 15:11:03 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/09 15:11:08 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/09 15:11:09 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:11:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:11:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:11:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:11:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:11:14 Rebalance progress: 12.5
2021/12/09 15:11:19 Rebalance progress: 12.5
2021/12/09 15:11:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:11:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:11:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:11:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 15:11:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 15:11:29 Rebalance progress: 12.5
2021/12/09 15:11:34 Rebalance progress: 12.5
2021/12/09 15:11:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:11:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:11:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:11:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 15:11:44 Rebalance progress: 12.5
2021/12/09 15:11:49 Rebalance progress: 12.5
2021/12/09 15:11:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:11:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:11:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:11:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:11:59 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 15:12:04 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 15:12:09 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 15:12:14 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 15:12:14 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/09 15:12:14 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:12:14 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/09 15:12:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:12:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:12:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7576
2021/12/09 15:12:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:12:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:12:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:12:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7576
2021/12/09 15:12:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7576
2021/12/09 15:12:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:12:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:12:51 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 15:12:56 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 15:13:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 15:13:06 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 15:13:06 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/09 15:13:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:13:06 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/09 15:13:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:13:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:13:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:13:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:13:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:13:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.69s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/09 15:13:30 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 15:13:30 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 15:13:30 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:13:30 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/09 15:13:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:13:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:13:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:13:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:13:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:13:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:13:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:13:48 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:13:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/09 15:13:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 15:13:59 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 15:14:04 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 15:14:09 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 15:14:09 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:14:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:14:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:14:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:14:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:14:30 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:14:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:14:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:15:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:15:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:15:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:15:06 Rebalance progress: 12.5
2021/12/09 15:15:11 Rebalance progress: 12.5
2021/12/09 15:15:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:15:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:15:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:15:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 15:15:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 15:15:21 Rebalance progress: 12.5
2021/12/09 15:15:26 Rebalance progress: 12.5
2021/12/09 15:15:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:15:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:15:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:15:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 15:15:36 Rebalance progress: 12.5
2021/12/09 15:15:41 Rebalance progress: 12.5
2021/12/09 15:15:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:15:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:15:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:15:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:15:51 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 15:15:56 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 15:16:01 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 15:16:06 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 15:16:06 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 15:16:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:16:06 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/09 15:16:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:16:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:16:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11185
2021/12/09 15:16:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:16:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:16:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:16:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 955
2021/12/09 15:16:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 955
2021/12/09 15:16:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.25s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/09 15:16:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 15:16:45 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 15:16:45 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:16:45 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/09 15:16:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:16:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:16:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:16:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:16:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:16:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:17:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:17:03 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:17:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/09 15:17:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 15:17:15 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 15:17:20 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 15:17:25 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 15:17:25 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:17:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:17:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:17:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:17:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:17:45 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:17:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:17:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:17:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:17:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:17:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:17:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:18:16 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:18:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:18:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:18:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:18:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:18:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:18:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:18:47 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:18:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:18:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:18:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:18:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:18:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:18:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:19:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:19:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:19:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:19:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:19:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:19:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:19:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:19:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:19:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:19:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:20:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:20:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:20:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:20:05 Rebalance progress: 12.5
2021/12/09 15:20:10 Rebalance progress: 12.5
2021/12/09 15:20:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:20:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:20:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:20:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 15:20:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 15:20:20 Rebalance progress: 12.5
2021/12/09 15:20:25 Rebalance progress: 12.5
2021/12/09 15:20:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:20:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:20:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:20:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 15:20:36 Rebalance progress: 12.5
2021/12/09 15:20:41 Rebalance progress: 12.5
2021/12/09 15:20:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:20:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:20:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:20:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:20:51 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 15:20:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 15:21:01 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 15:21:06 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 15:21:06 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 15:21:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:21:06 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/09 15:21:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:21:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:21:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21577
2021/12/09 15:21:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:21:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:21:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:21:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (289.37s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/09 15:21:34 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 15:21:34 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/09 15:21:34 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:21:34 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/09 15:21:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:21:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:21:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:21:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:21:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:21:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:21:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:21:53 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:21:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 15:21:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 15:22:04 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/09 15:22:09 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/09 15:22:14 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/09 15:22:14 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:22:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:22:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:22:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:22:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:22:20 Rebalance progress: 12.5
2021/12/09 15:22:25 Rebalance progress: 12.5
2021/12/09 15:22:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:22:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:22:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:22:30 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:22:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:22:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:22:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:22:45 Rebalance progress: 12.5
2021/12/09 15:22:50 Rebalance progress: 12.5
2021/12/09 15:22:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:22:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:22:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:22:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 15:22:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 15:23:00 Rebalance progress: 12.5
2021/12/09 15:23:05 Rebalance progress: 12.5
2021/12/09 15:23:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:23:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:23:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:23:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 15:23:15 Rebalance progress: 12.5
2021/12/09 15:23:20 Rebalance progress: 12.5
2021/12/09 15:23:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:23:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:23:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:23:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:23:30 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 15:23:35 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 15:23:40 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 15:23:45 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 15:23:45 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/09 15:23:45 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:23:45 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/09 15:23:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:23:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:23:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8615
2021/12/09 15:23:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:23:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:24:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:24:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8615
2021/12/09 15:24:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (164.57s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/09 15:24:19 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 15:24:19 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/09 15:24:19 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:24:19 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/09 15:24:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:24:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:24:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:24:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:24:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:24:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:24:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:24:37 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:24:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 15:24:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 15:24:49 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/09 15:24:54 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/09 15:24:59 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/09 15:24:59 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:24:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:24:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:24:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/09 15:24:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:25:04 Rebalance progress: 12.5
2021/12/09 15:25:09 Rebalance progress: 12.5
2021/12/09 15:25:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:25:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:25:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:25:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/09 15:25:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 15:25:20 Rebalance progress: 12.5
2021/12/09 15:25:25 Rebalance progress: 12.5
2021/12/09 15:25:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:25:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:25:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:25:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 15:25:35 Rebalance progress: 12.5
2021/12/09 15:25:40 Rebalance progress: 12.5
2021/12/09 15:25:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:25:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:25:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:25:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:25:50 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 15:25:55 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 15:26:00 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 15:26:05 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 15:26:05 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/09 15:26:05 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:26:05 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/09 15:26:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:26:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:26:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6096
2021/12/09 15:26:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:26:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:26:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:26:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6096
2021/12/09 15:26:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.51s)
=== RUN   TestEventingRebInWithOps
2021/12/09 15:26:38 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 15:26:38 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/09 15:26:38 eventingSetup.go:46: No eventing-con process running
2021/12/09 15:26:38 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/09 15:26:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:26:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:26:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:26:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:26:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:26:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:26:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:26:57 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:26:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 15:27:03 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 15:27:08 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 15:27:13 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 15:27:18 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 15:27:18 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/09 15:27:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:27:18 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 15:27:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 15:27:23 Rebalance progress: 12.5
2021/12/09 15:27:28 Rebalance progress: 12.5
2021/12/09 15:27:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:27:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:27:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:27:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 15:27:39 Rebalance progress: 12.5
2021/12/09 15:27:44 Rebalance progress: 12.5
2021/12/09 15:27:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:27:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:27:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:27:49 Waiting for src bucket item count to get to: 200000 curr count: 32481
2021/12/09 15:27:54 Waiting for src bucket item count to get to: 200000 curr count: 48814
2021/12/09 15:27:59 Waiting for src bucket item count to get to: 200000 curr count: 48814
2021/12/09 15:28:04 Waiting for src bucket item count to get to: 200000 curr count: 65340
2021/12/09 15:28:09 Waiting for src bucket item count to get to: 200000 curr count: 65340
2021/12/09 15:28:14 Waiting for src bucket item count to get to: 200000 curr count: 80517
2021/12/09 15:28:19 Waiting for src bucket item count to get to: 200000 curr count: 80517
2021/12/09 15:28:24 Waiting for src bucket item count to get to: 200000 curr count: 96701
2021/12/09 15:28:29 Waiting for src bucket item count to get to: 200000 curr count: 96701
2021/12/09 15:28:34 Waiting for src bucket item count to get to: 200000 curr count: 96701
2021/12/09 15:28:39 Waiting for src bucket item count to get to: 200000 curr count: 113402
2021/12/09 15:28:44 Waiting for src bucket item count to get to: 200000 curr count: 113402
2021/12/09 15:28:49 Waiting for src bucket item count to get to: 200000 curr count: 129981
2021/12/09 15:28:54 Waiting for src bucket item count to get to: 200000 curr count: 129981
2021/12/09 15:28:59 Waiting for src bucket item count to get to: 200000 curr count: 146146
2021/12/09 15:29:04 Waiting for src bucket item count to get to: 200000 curr count: 146146
2021/12/09 15:29:09 Waiting for src bucket item count to get to: 200000 curr count: 162573
2021/12/09 15:29:14 Waiting for src bucket item count to get to: 200000 curr count: 162573
2021/12/09 15:29:19 Waiting for src bucket item count to get to: 200000 curr count: 177872
2021/12/09 15:29:24 Waiting for src bucket item count to get to: 200000 curr count: 177872
2021/12/09 15:29:29 Waiting for src bucket item count to get to: 200000 curr count: 194068
2021/12/09 15:29:34 Waiting for src bucket item count to get to: 200000 curr count: 194068
2021/12/09 15:29:39 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 15:29:39 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 15:29:39 Starting bucket ops to source bucket
2021/12/09 15:33:24 Waiting for dst bucket item count to get to: 0 curr count: 17594
2021/12/09 15:33:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:33:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 15:33:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:33:29 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 15:33:34 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 15:33:39 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 15:33:44 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 15:33:44 App: TestEventingRebInWithOps got un-deployed
2021/12/09 15:33:44 No eventing-con process running
2021/12/09 15:33:44 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/09 15:33:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:33:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:33:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:33:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:33:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:33:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (449.09s)
=== RUN   TestEventingRebOutWithOps
2021/12/09 15:34:03 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:34:03 Removing node(s): [] from the cluster
2021/12/09 15:34:08 Rebalance progress: 100
2021/12/09 15:34:13 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 15:34:13 App: TestEventingRebOutWithOps got un-deployed
2021/12/09 15:34:13 No eventing-con process running
2021/12/09 15:34:13 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/09 15:34:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:34:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:34:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:34:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:34:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:34:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:34:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:34:32 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:34:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 15:34:38 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 15:34:43 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 15:34:48 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 15:34:53 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 15:34:53 App: TestEventingRebOutWithOps got deployed
2021/12/09 15:34:53 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 15:34:53 Starting bucket ops to source bucket
2021/12/09 15:34:58 Rebalance progress: 12.5
2021/12/09 15:35:03 Rebalance progress: 12.5
2021/12/09 15:35:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:35:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:35:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:35:08 Waiting for src bucket item count to get to: 200000 curr count: 9339
2021/12/09 15:35:13 Waiting for src bucket item count to get to: 200000 curr count: 25505
2021/12/09 15:35:18 Waiting for src bucket item count to get to: 200000 curr count: 25505
2021/12/09 15:35:23 Waiting for src bucket item count to get to: 200000 curr count: 41909
2021/12/09 15:35:28 Waiting for src bucket item count to get to: 200000 curr count: 41909
2021/12/09 15:35:33 Waiting for src bucket item count to get to: 200000 curr count: 41909
2021/12/09 15:35:38 Waiting for src bucket item count to get to: 200000 curr count: 58374
2021/12/09 15:35:43 Waiting for src bucket item count to get to: 200000 curr count: 58374
2021/12/09 15:35:48 Waiting for src bucket item count to get to: 200000 curr count: 74049
2021/12/09 15:35:53 Waiting for src bucket item count to get to: 200000 curr count: 74049
2021/12/09 15:35:58 Waiting for src bucket item count to get to: 200000 curr count: 90079
2021/12/09 15:36:03 Waiting for src bucket item count to get to: 200000 curr count: 90079
2021/12/09 15:36:08 Waiting for src bucket item count to get to: 200000 curr count: 106648
2021/12/09 15:36:13 Waiting for src bucket item count to get to: 200000 curr count: 106648
2021/12/09 15:36:18 Waiting for src bucket item count to get to: 200000 curr count: 123183
2021/12/09 15:36:23 Waiting for src bucket item count to get to: 200000 curr count: 123183
2021/12/09 15:36:28 Waiting for src bucket item count to get to: 200000 curr count: 139751
2021/12/09 15:36:33 Waiting for src bucket item count to get to: 200000 curr count: 139751
2021/12/09 15:36:38 Waiting for src bucket item count to get to: 200000 curr count: 156364
2021/12/09 15:36:43 Waiting for src bucket item count to get to: 200000 curr count: 156364
2021/12/09 15:36:48 Waiting for src bucket item count to get to: 200000 curr count: 172581
2021/12/09 15:36:53 Waiting for src bucket item count to get to: 200000 curr count: 172581
2021/12/09 15:36:58 Waiting for src bucket item count to get to: 200000 curr count: 188320
2021/12/09 15:37:03 Waiting for src bucket item count to get to: 200000 curr count: 188320
2021/12/09 15:37:08 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 15:37:08 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 15:37:08 Starting bucket ops to source bucket
2021/12/09 15:40:53 Waiting for dst bucket item count to get to: 0 curr count: 15987
2021/12/09 15:40:58 Waiting for dst bucket item count to get to: 0 curr count: 15987
2021/12/09 15:40:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:40:58 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 15:40:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:41:03 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 15:41:08 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 15:41:13 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 15:41:18 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 15:41:18 App: TestEventingRebOutWithOps got un-deployed
2021/12/09 15:41:18 No eventing-con process running
2021/12/09 15:41:18 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/09 15:41:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:41:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:41:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:41:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:41:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:41:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (454.35s)
=== RUN   TestEventingSwapRebWithOps
2021/12/09 15:41:42 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 15:41:42 App: TestEventingSwapRebWithOps got un-deployed
2021/12/09 15:41:42 No eventing-con process running
2021/12/09 15:41:42 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/09 15:41:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:41:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:41:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:41:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:41:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:41:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:42:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:42:00 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:42:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 15:42:06 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 15:42:11 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 15:42:17 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 15:42:22 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 15:42:22 App: TestEventingSwapRebWithOps got deployed
2021/12/09 15:42:22 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 15:42:22 Starting bucket ops to source bucket
2021/12/09 15:42:22 Removing node(s): [] from the cluster
2021/12/09 15:42:27 Rebalance progress: 12.5
2021/12/09 15:42:32 Rebalance progress: 12.5
2021/12/09 15:42:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:42:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:42:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:42:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 15:42:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 15:42:42 Rebalance progress: 12.5
2021/12/09 15:42:47 Rebalance progress: 12.5
2021/12/09 15:42:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:42:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:42:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:42:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 15:42:57 Rebalance progress: 12.5
2021/12/09 15:43:02 Rebalance progress: 12.5
2021/12/09 15:43:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:43:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:43:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:43:07 Waiting for src bucket item count to get to: 200000 curr count: 59086
2021/12/09 15:43:12 Waiting for src bucket item count to get to: 200000 curr count: 74207
2021/12/09 15:43:17 Waiting for src bucket item count to get to: 200000 curr count: 74207
2021/12/09 15:43:22 Waiting for src bucket item count to get to: 200000 curr count: 90486
2021/12/09 15:43:27 Waiting for src bucket item count to get to: 200000 curr count: 90486
2021/12/09 15:43:32 Waiting for src bucket item count to get to: 200000 curr count: 106892
2021/12/09 15:43:37 Waiting for src bucket item count to get to: 200000 curr count: 106892
2021/12/09 15:43:42 Waiting for src bucket item count to get to: 200000 curr count: 122913
2021/12/09 15:43:47 Waiting for src bucket item count to get to: 200000 curr count: 122913
2021/12/09 15:43:52 Waiting for src bucket item count to get to: 200000 curr count: 139273
2021/12/09 15:43:57 Waiting for src bucket item count to get to: 200000 curr count: 139273
2021/12/09 15:44:02 Waiting for src bucket item count to get to: 200000 curr count: 139273
2021/12/09 15:44:07 Waiting for src bucket item count to get to: 200000 curr count: 155561
2021/12/09 15:44:12 Waiting for src bucket item count to get to: 200000 curr count: 155561
2021/12/09 15:44:17 Waiting for src bucket item count to get to: 200000 curr count: 170885
2021/12/09 15:44:22 Waiting for src bucket item count to get to: 200000 curr count: 170885
2021/12/09 15:44:27 Waiting for src bucket item count to get to: 200000 curr count: 186632
2021/12/09 15:44:32 Waiting for src bucket item count to get to: 200000 curr count: 186632
2021/12/09 15:44:37 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 15:44:37 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 15:44:37 Starting bucket ops to source bucket
2021/12/09 15:48:23 Waiting for dst bucket item count to get to: 0 curr count: 15252
2021/12/09 15:48:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:48:23 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 15:48:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:48:28 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 15:48:33 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 15:48:38 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 15:48:43 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 15:48:43 App: TestEventingSwapRebWithOps got un-deployed
2021/12/09 15:48:43 No eventing-con process running
2021/12/09 15:48:43 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/09 15:48:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:48:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:48:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:48:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:48:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:48:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (444.40s)
=== RUN   TestKVRebInWithOps
2021/12/09 15:49:06 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 15:49:06 App: TestKVRebInWithOps got un-deployed
2021/12/09 15:49:06 No eventing-con process running
2021/12/09 15:49:06 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/09 15:49:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:49:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:49:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:49:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:49:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:49:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:49:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:49:25 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:49:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 15:49:31 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 15:49:36 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 15:49:41 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 15:49:46 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 15:49:46 App: TestKVRebInWithOps got deployed
2021/12/09 15:49:46 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 15:49:46 Starting bucket ops to source bucket
2021/12/09 15:49:46 Removing node(s): [] from the cluster
2021/12/09 15:49:51 Rebalance progress: 12.5
2021/12/09 15:49:56 Rebalance progress: 12.5
2021/12/09 15:50:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:50:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:50:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:50:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 15:50:06 Rebalance progress: 12.5
2021/12/09 15:50:11 Rebalance progress: 12.5
2021/12/09 15:50:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:50:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:50:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:50:16 Waiting for src bucket item count to get to: 200000 curr count: 36355
2021/12/09 15:50:21 Waiting for src bucket item count to get to: 200000 curr count: 52612
2021/12/09 15:50:26 Waiting for src bucket item count to get to: 200000 curr count: 52612
2021/12/09 15:50:31 Waiting for src bucket item count to get to: 200000 curr count: 69188
2021/12/09 15:50:36 Waiting for src bucket item count to get to: 200000 curr count: 69188
2021/12/09 15:50:41 Waiting for src bucket item count to get to: 200000 curr count: 84364
2021/12/09 15:50:46 Waiting for src bucket item count to get to: 200000 curr count: 84364
2021/12/09 15:50:51 Waiting for src bucket item count to get to: 200000 curr count: 100876
2021/12/09 15:50:56 Waiting for src bucket item count to get to: 200000 curr count: 100876
2021/12/09 15:51:01 Waiting for src bucket item count to get to: 200000 curr count: 100876
2021/12/09 15:51:06 Waiting for src bucket item count to get to: 200000 curr count: 117205
2021/12/09 15:51:12 Waiting for src bucket item count to get to: 200000 curr count: 117205
2021/12/09 15:51:17 Waiting for src bucket item count to get to: 200000 curr count: 133627
2021/12/09 15:51:22 Waiting for src bucket item count to get to: 200000 curr count: 133627
2021/12/09 15:51:27 Waiting for src bucket item count to get to: 200000 curr count: 149881
2021/12/09 15:51:32 Waiting for src bucket item count to get to: 200000 curr count: 149881
2021/12/09 15:51:37 Waiting for src bucket item count to get to: 200000 curr count: 166389
2021/12/09 15:51:42 Waiting for src bucket item count to get to: 200000 curr count: 166389
2021/12/09 15:51:47 Waiting for src bucket item count to get to: 200000 curr count: 181777
2021/12/09 15:51:52 Waiting for src bucket item count to get to: 200000 curr count: 181777
2021/12/09 15:51:57 Waiting for src bucket item count to get to: 200000 curr count: 198151
2021/12/09 15:52:02 Waiting for src bucket item count to get to: 200000 curr count: 198151
2021/12/09 15:52:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 15:52:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 15:52:07 Starting bucket ops to source bucket
2021/12/09 15:55:51 Waiting for dst bucket item count to get to: 0 curr count: 13442
2021/12/09 15:55:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:55:51 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 15:55:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 15:55:57 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 15:56:02 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 15:56:07 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 15:56:12 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 15:56:12 App: TestKVRebInWithOps got un-deployed
2021/12/09 15:56:12 No eventing-con process running
2021/12/09 15:56:12 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/09 15:56:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:56:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:56:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:56:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:56:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:56:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (448.88s)
=== RUN   TestKVRebOutWithOps
2021/12/09 15:56:30 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 15:56:30 Removing node(s): [] from the cluster
2021/12/09 15:56:35 Rebalance progress: 100
2021/12/09 15:56:40 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 15:56:40 App: TestKVRebOutWithOps got un-deployed
2021/12/09 15:56:40 No eventing-con process running
2021/12/09 15:56:40 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/09 15:56:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:56:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:56:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 15:56:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:56:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:56:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:56:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 15:56:59 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 15:57:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 15:57:05 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 15:57:10 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 15:57:15 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 15:57:20 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 15:57:20 App: TestKVRebOutWithOps got deployed
2021/12/09 15:57:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 15:57:20 Starting bucket ops to source bucket
2021/12/09 15:57:25 Rebalance progress: 12.5
2021/12/09 15:57:30 Rebalance progress: 12.5
2021/12/09 15:57:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:57:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 15:57:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 15:57:35 Waiting for src bucket item count to get to: 200000 curr count: 14261
2021/12/09 15:57:40 Waiting for src bucket item count to get to: 200000 curr count: 30433
2021/12/09 15:57:45 Waiting for src bucket item count to get to: 200000 curr count: 30433
2021/12/09 15:57:50 Waiting for src bucket item count to get to: 200000 curr count: 46994
2021/12/09 15:57:55 Waiting for src bucket item count to get to: 200000 curr count: 46994
2021/12/09 15:58:00 Waiting for src bucket item count to get to: 200000 curr count: 63372
2021/12/09 15:58:05 Waiting for src bucket item count to get to: 200000 curr count: 63372
2021/12/09 15:58:10 Waiting for src bucket item count to get to: 200000 curr count: 63372
2021/12/09 15:58:15 Waiting for src bucket item count to get to: 200000 curr count: 78918
2021/12/09 15:58:20 Waiting for src bucket item count to get to: 200000 curr count: 78918
2021/12/09 15:58:25 Waiting for src bucket item count to get to: 200000 curr count: 94999
2021/12/09 15:58:30 Waiting for src bucket item count to get to: 200000 curr count: 94999
2021/12/09 15:58:35 Waiting for src bucket item count to get to: 200000 curr count: 111420
2021/12/09 15:58:40 Waiting for src bucket item count to get to: 200000 curr count: 111420
2021/12/09 15:58:45 Waiting for src bucket item count to get to: 200000 curr count: 128115
2021/12/09 15:58:50 Waiting for src bucket item count to get to: 200000 curr count: 128115
2021/12/09 15:58:55 Waiting for src bucket item count to get to: 200000 curr count: 144495
2021/12/09 15:59:00 Waiting for src bucket item count to get to: 200000 curr count: 144495
2021/12/09 15:59:05 Waiting for src bucket item count to get to: 200000 curr count: 161107
2021/12/09 15:59:10 Waiting for src bucket item count to get to: 200000 curr count: 161107
2021/12/09 15:59:15 Waiting for src bucket item count to get to: 200000 curr count: 176623
2021/12/09 15:59:20 Waiting for src bucket item count to get to: 200000 curr count: 176623
2021/12/09 15:59:25 Waiting for src bucket item count to get to: 200000 curr count: 192682
2021/12/09 15:59:30 Waiting for src bucket item count to get to: 200000 curr count: 192682
2021/12/09 15:59:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 15:59:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 15:59:35 Starting bucket ops to source bucket
2021/12/09 16:03:20 Waiting for dst bucket item count to get to: 0 curr count: 11014
2021/12/09 16:03:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:03:20 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 16:03:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:03:25 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 16:03:30 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 16:03:35 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 16:03:40 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 16:03:41 App: TestKVRebOutWithOps got un-deployed
2021/12/09 16:03:41 No eventing-con process running
2021/12/09 16:03:41 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/09 16:03:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:03:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:03:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:03:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:03:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:03:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (448.90s)
=== RUN   TestKVSwapRebWithOps
2021/12/09 16:04:04 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 16:04:04 App: TestKVSwapRebWithOps got un-deployed
2021/12/09 16:04:04 No eventing-con process running
2021/12/09 16:04:04 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/09 16:04:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:04:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:04:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:04:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:04:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:04:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:04:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:04:23 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:04:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 16:04:29 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 16:04:34 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 16:04:39 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 16:04:44 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 16:04:44 App: TestKVSwapRebWithOps got deployed
2021/12/09 16:04:44 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 16:04:44 Starting bucket ops to source bucket
2021/12/09 16:04:44 Removing node(s): [] from the cluster
2021/12/09 16:04:49 Rebalance progress: 12.5
2021/12/09 16:04:54 Rebalance progress: 12.5
2021/12/09 16:04:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:04:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:04:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:04:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 16:05:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:05:05 Rebalance progress: 12.5
2021/12/09 16:05:10 Rebalance progress: 12.5
2021/12/09 16:05:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:05:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:05:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:05:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 16:05:20 Rebalance progress: 12.5
2021/12/09 16:05:25 Rebalance progress: 12.5
2021/12/09 16:05:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:05:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:05:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:05:30 Waiting for src bucket item count to get to: 200000 curr count: 55173
2021/12/09 16:05:35 Waiting for src bucket item count to get to: 200000 curr count: 70993
2021/12/09 16:05:40 Waiting for src bucket item count to get to: 200000 curr count: 70993
2021/12/09 16:05:45 Waiting for src bucket item count to get to: 200000 curr count: 86056
2021/12/09 16:05:50 Waiting for src bucket item count to get to: 200000 curr count: 86056
2021/12/09 16:05:55 Waiting for src bucket item count to get to: 200000 curr count: 102599
2021/12/09 16:06:00 Waiting for src bucket item count to get to: 200000 curr count: 102599
2021/12/09 16:06:05 Waiting for src bucket item count to get to: 200000 curr count: 119025
2021/12/09 16:06:10 Waiting for src bucket item count to get to: 200000 curr count: 119025
2021/12/09 16:06:15 Waiting for src bucket item count to get to: 200000 curr count: 135338
2021/12/09 16:06:20 Waiting for src bucket item count to get to: 200000 curr count: 151902
2021/12/09 16:06:25 Waiting for src bucket item count to get to: 200000 curr count: 151902
2021/12/09 16:06:30 Waiting for src bucket item count to get to: 200000 curr count: 168240
2021/12/09 16:06:35 Waiting for src bucket item count to get to: 200000 curr count: 168240
2021/12/09 16:06:40 Waiting for src bucket item count to get to: 200000 curr count: 168240
2021/12/09 16:06:45 Waiting for src bucket item count to get to: 200000 curr count: 183541
2021/12/09 16:06:50 Waiting for src bucket item count to get to: 200000 curr count: 183541
2021/12/09 16:06:55 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 16:06:55 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 16:06:55 Starting bucket ops to source bucket
2021/12/09 16:10:40 Waiting for dst bucket item count to get to: 0 curr count: 9810
2021/12/09 16:10:45 Waiting for dst bucket item count to get to: 0 curr count: 9810
2021/12/09 16:10:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:10:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 16:10:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:10:50 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 16:10:55 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 16:11:00 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 16:11:05 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 16:11:05 App: TestKVSwapRebWithOps got un-deployed
2021/12/09 16:11:05 No eventing-con process running
2021/12/09 16:11:05 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/09 16:11:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:11:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:11:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:11:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:11:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:11:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (444.63s)
=== RUN   TestEventingRebInWithTimers
2021/12/09 16:11:29 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 16:11:29 App: TestEventingRebInWithTimers got un-deployed
2021/12/09 16:11:29 No eventing-con process running
2021/12/09 16:11:29 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/09 16:11:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:11:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:11:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:11:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:11:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:11:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:11:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:11:47 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:11:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 16:11:53 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 16:11:58 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 16:12:03 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 16:12:08 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 16:12:08 App: TestEventingRebInWithTimers got deployed
2021/12/09 16:12:08 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 16:12:08 Starting bucket ops to source bucket
2021/12/09 16:12:09 Removing node(s): [] from the cluster
2021/12/09 16:12:14 Rebalance progress: 12.5
2021/12/09 16:12:19 Rebalance progress: 12.5
2021/12/09 16:12:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:12:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:12:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:12:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:12:29 Rebalance progress: 12.5
2021/12/09 16:12:34 Rebalance progress: 12.5
2021/12/09 16:12:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:12:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:12:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:12:39 Waiting for src bucket item count to get to: 50000 curr count: 25941
2021/12/09 16:12:44 Waiting for src bucket item count to get to: 50000 curr count: 36662
2021/12/09 16:12:49 Waiting for src bucket item count to get to: 50000 curr count: 36662
2021/12/09 16:12:54 Waiting for src bucket item count to get to: 50000 curr count: 49365
2021/12/09 16:12:59 Waiting for src bucket item count to get to: 50000 curr count: 49365
2021/12/09 16:13:09 Waiting for dst bucket item count to get to: 50000 curr count: 49953
2021/12/09 16:13:14 Waiting for dst bucket item count to get to: 50000 curr count: 49953
2021/12/09 16:13:14 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:13:14 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:13:14 Starting bucket ops to source bucket
2021/12/09 16:14:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:14:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:14:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:14:39 Waiting for dst bucket item count to get to: 0 curr count: 36794
2021/12/09 16:14:44 Waiting for dst bucket item count to get to: 0 curr count: 36794
2021/12/09 16:14:49 Waiting for dst bucket item count to get to: 0 curr count: 22377
2021/12/09 16:14:54 Waiting for dst bucket item count to get to: 0 curr count: 22377
2021/12/09 16:14:59 Waiting for dst bucket item count to get to: 0 curr count: 8103
2021/12/09 16:15:04 Waiting for dst bucket item count to get to: 0 curr count: 8103
2021/12/09 16:15:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:15:04 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 16:15:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:15:09 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 16:15:14 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 16:15:19 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 16:15:24 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 16:15:24 App: TestEventingRebInWithTimers got un-deployed
2021/12/09 16:15:24 No eventing-con process running
2021/12/09 16:15:24 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/09 16:15:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:15:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:15:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:15:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:15:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:15:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (258.84s)
=== RUN   TestEventingRebOutWithTimers
2021/12/09 16:15:43 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 16:15:43 Removing node(s): [] from the cluster
2021/12/09 16:15:48 Rebalance progress: 100
2021/12/09 16:15:53 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 16:15:53 App: TestEventingRebOutWithTimers got un-deployed
2021/12/09 16:15:53 No eventing-con process running
2021/12/09 16:15:53 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/09 16:15:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:16:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:16:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:16:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:16:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:16:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:16:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:16:12 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:16:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 16:16:18 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 16:16:23 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 16:16:28 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 16:16:33 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 16:16:33 App: TestEventingRebOutWithTimers got deployed
2021/12/09 16:16:33 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:16:33 Starting bucket ops to source bucket
2021/12/09 16:16:38 Rebalance progress: 12.5
2021/12/09 16:16:43 Rebalance progress: 12.5
2021/12/09 16:16:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:16:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:16:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:16:48 Waiting for src bucket item count to get to: 50000 curr count: 9059
2021/12/09 16:16:53 Waiting for src bucket item count to get to: 50000 curr count: 23597
2021/12/09 16:16:58 Waiting for src bucket item count to get to: 50000 curr count: 23597
2021/12/09 16:17:03 Waiting for src bucket item count to get to: 50000 curr count: 35495
2021/12/09 16:17:08 Waiting for src bucket item count to get to: 50000 curr count: 35495
2021/12/09 16:17:13 Waiting for src bucket item count to get to: 50000 curr count: 45978
2021/12/09 16:17:18 Waiting for src bucket item count to get to: 50000 curr count: 45978
2021/12/09 16:17:23 Waiting for src bucket item count to get to: 50000 curr count: 45978
2021/12/09 16:17:33 Waiting for dst bucket item count to get to: 50000 curr count: 36594
2021/12/09 16:17:38 Waiting for dst bucket item count to get to: 50000 curr count: 36594
2021/12/09 16:17:43 Waiting for dst bucket item count to get to: 50000 curr count: 49878
2021/12/09 16:17:48 Waiting for dst bucket item count to get to: 50000 curr count: 49878
2021/12/09 16:17:48 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:17:48 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:17:48 Starting bucket ops to source bucket
2021/12/09 16:18:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:19:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:19:08 Waiting for dst bucket item count to get to: 0 curr count: 48847
2021/12/09 16:19:13 Waiting for dst bucket item count to get to: 0 curr count: 48847
2021/12/09 16:19:18 Waiting for dst bucket item count to get to: 0 curr count: 34830
2021/12/09 16:19:23 Waiting for dst bucket item count to get to: 0 curr count: 34830
2021/12/09 16:19:28 Waiting for dst bucket item count to get to: 0 curr count: 24513
2021/12/09 16:19:33 Waiting for dst bucket item count to get to: 0 curr count: 24513
2021/12/09 16:19:38 Waiting for dst bucket item count to get to: 0 curr count: 3100
2021/12/09 16:19:43 Waiting for dst bucket item count to get to: 0 curr count: 3100
2021/12/09 16:19:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:19:43 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 16:19:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:19:48 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 16:19:53 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 16:19:58 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 16:20:03 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 16:20:03 App: TestEventingRebOutWithTimers got un-deployed
2021/12/09 16:20:03 No eventing-con process running
2021/12/09 16:20:03 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/09 16:20:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:20:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:20:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:20:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:20:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:20:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (278.98s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/09 16:20:26 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 16:20:27 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/09 16:20:27 No eventing-con process running
2021/12/09 16:20:27 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/09 16:20:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:20:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:20:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:20:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:20:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:20:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:20:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:20:45 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:20:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 16:20:51 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 16:20:56 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 16:21:01 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 16:21:06 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 16:21:06 App: TestEventingSwapRebWithTimers got deployed
2021/12/09 16:21:06 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 16:21:06 Starting bucket ops to source bucket
2021/12/09 16:21:07 Removing node(s): [] from the cluster
2021/12/09 16:21:12 Rebalance progress: 12.5
2021/12/09 16:21:17 Rebalance progress: 12.5
2021/12/09 16:21:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:21:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:21:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:21:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 16:21:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:21:28 Rebalance progress: 12.5
2021/12/09 16:21:33 Rebalance progress: 12.5
2021/12/09 16:21:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:21:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:21:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:21:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 16:21:43 Rebalance progress: 12.5
2021/12/09 16:21:48 Rebalance progress: 12.5
2021/12/09 16:21:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:21:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:21:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:21:58 Waiting for dst bucket item count to get to: 50000 curr count: 34662
2021/12/09 16:22:03 Waiting for dst bucket item count to get to: 50000 curr count: 34662
2021/12/09 16:22:08 Waiting for dst bucket item count to get to: 50000 curr count: 41355
2021/12/09 16:22:13 Waiting for dst bucket item count to get to: 50000 curr count: 41355
2021/12/09 16:22:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:22:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:22:13 Starting bucket ops to source bucket
2021/12/09 16:23:22 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:23:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:23:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:23:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:23:42 Waiting for dst bucket item count to get to: 0 curr count: 41489
2021/12/09 16:23:47 Waiting for dst bucket item count to get to: 0 curr count: 41489
2021/12/09 16:23:52 Waiting for dst bucket item count to get to: 0 curr count: 31964
2021/12/09 16:23:57 Waiting for dst bucket item count to get to: 0 curr count: 31964
2021/12/09 16:24:02 Waiting for dst bucket item count to get to: 0 curr count: 10321
2021/12/09 16:24:07 Waiting for dst bucket item count to get to: 0 curr count: 10321
2021/12/09 16:24:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:24:07 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 16:24:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:24:12 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 16:24:17 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 16:24:22 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 16:24:27 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 16:24:27 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/09 16:24:27 No eventing-con process running
2021/12/09 16:24:27 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/09 16:24:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:24:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:24:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:24:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:24:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:24:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (264.47s)
=== RUN   TestKVRebInWithTimers
2021/12/09 16:24:51 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 16:24:51 App: TestKVRebInWithTimers got un-deployed
2021/12/09 16:24:51 No eventing-con process running
2021/12/09 16:24:51 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/09 16:24:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:24:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:24:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:25:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:25:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:25:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:25:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:25:10 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:25:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 16:25:16 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 16:25:21 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 16:25:26 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 16:25:31 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 16:25:31 App: TestKVRebInWithTimers got deployed
2021/12/09 16:25:31 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 16:25:31 Starting bucket ops to source bucket
2021/12/09 16:25:31 Removing node(s): [] from the cluster
2021/12/09 16:25:36 Rebalance progress: 12.5
2021/12/09 16:25:41 Rebalance progress: 12.5
2021/12/09 16:25:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:25:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:25:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:25:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:25:51 Rebalance progress: 12.5
2021/12/09 16:25:56 Rebalance progress: 12.5
2021/12/09 16:26:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:26:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:26:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:26:01 Waiting for src bucket item count to get to: 50000 curr count: 35475
2021/12/09 16:26:06 Waiting for src bucket item count to get to: 50000 curr count: 35475
2021/12/09 16:26:11 Waiting for src bucket item count to get to: 50000 curr count: 46736
2021/12/09 16:26:16 Waiting for src bucket item count to get to: 50000 curr count: 46736
2021/12/09 16:26:26 Waiting for dst bucket item count to get to: 50000 curr count: 38020
2021/12/09 16:26:31 Waiting for dst bucket item count to get to: 50000 curr count: 38020
2021/12/09 16:26:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:26:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:26:31 Starting bucket ops to source bucket
2021/12/09 16:27:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:27:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:27:50 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:27:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:28:00 Waiting for dst bucket item count to get to: 0 curr count: 42853
2021/12/09 16:28:05 Waiting for dst bucket item count to get to: 0 curr count: 42853
2021/12/09 16:28:10 Waiting for dst bucket item count to get to: 0 curr count: 22472
2021/12/09 16:28:15 Waiting for dst bucket item count to get to: 0 curr count: 22472
2021/12/09 16:28:20 Waiting for dst bucket item count to get to: 0 curr count: 8583
2021/12/09 16:28:25 Waiting for dst bucket item count to get to: 0 curr count: 8583
2021/12/09 16:28:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:28:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 16:28:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:28:31 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 16:28:36 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 16:28:41 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 16:28:46 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 16:28:46 App: TestKVRebInWithTimers got un-deployed
2021/12/09 16:28:46 No eventing-con process running
2021/12/09 16:28:46 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/09 16:28:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:28:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:28:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:28:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:28:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:28:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (258.22s)
=== RUN   TestKVRebOutWithTimers
2021/12/09 16:29:04 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 16:29:04 Removing node(s): [] from the cluster
2021/12/09 16:29:10 Rebalance progress: 100
2021/12/09 16:29:15 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 16:29:15 App: TestKVRebOutWithTimers got un-deployed
2021/12/09 16:29:15 No eventing-con process running
2021/12/09 16:29:15 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/09 16:29:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:29:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:29:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:29:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:29:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:29:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:29:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:29:33 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:29:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 16:29:39 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 16:29:44 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 16:29:49 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 16:29:54 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 16:29:54 App: TestKVRebOutWithTimers got deployed
2021/12/09 16:29:54 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:29:54 Starting bucket ops to source bucket
2021/12/09 16:29:59 Rebalance progress: 12.5
2021/12/09 16:30:04 Rebalance progress: 12.5
2021/12/09 16:30:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:30:09 Waiting for src bucket item count to get to: 50000 curr count: 6487
2021/12/09 16:30:14 Waiting for src bucket item count to get to: 50000 curr count: 19771
2021/12/09 16:30:19 Waiting for src bucket item count to get to: 50000 curr count: 19771
2021/12/09 16:30:24 Waiting for src bucket item count to get to: 50000 curr count: 30922
2021/12/09 16:30:29 Waiting for src bucket item count to get to: 50000 curr count: 30922
2021/12/09 16:30:34 Waiting for src bucket item count to get to: 50000 curr count: 42562
2021/12/09 16:30:39 Waiting for src bucket item count to get to: 50000 curr count: 42562
2021/12/09 16:30:44 Waiting for src bucket item count to get to: 50000 curr count: 42562
2021/12/09 16:30:54 Waiting for dst bucket item count to get to: 50000 curr count: 40503
2021/12/09 16:30:59 Waiting for dst bucket item count to get to: 50000 curr count: 48609
2021/12/09 16:31:04 Waiting for dst bucket item count to get to: 50000 curr count: 48609
2021/12/09 16:31:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:31:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:31:04 Starting bucket ops to source bucket
2021/12/09 16:32:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:32:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:32:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:32:29 Waiting for dst bucket item count to get to: 0 curr count: 47450
2021/12/09 16:32:34 Waiting for dst bucket item count to get to: 0 curr count: 47450
2021/12/09 16:32:39 Waiting for dst bucket item count to get to: 0 curr count: 29703
2021/12/09 16:32:44 Waiting for dst bucket item count to get to: 0 curr count: 29703
2021/12/09 16:32:49 Waiting for dst bucket item count to get to: 0 curr count: 9249
2021/12/09 16:32:54 Waiting for dst bucket item count to get to: 0 curr count: 9249
2021/12/09 16:32:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:32:54 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 16:32:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:32:59 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 16:33:04 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 16:33:09 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 16:33:14 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 16:33:14 App: TestKVRebOutWithTimers got un-deployed
2021/12/09 16:33:14 No eventing-con process running
2021/12/09 16:33:14 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/09 16:33:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:33:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:33:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:33:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:33:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (268.54s)
=== RUN   TestKVSwapRebWithTimers
2021/12/09 16:33:38 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 16:33:38 App: TestKVSwapRebWithTimers got un-deployed
2021/12/09 16:33:38 No eventing-con process running
2021/12/09 16:33:38 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/09 16:33:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:33:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:33:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:33:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:33:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:33:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:33:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:33:56 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:33:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 16:34:02 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 16:34:07 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 16:34:12 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 16:34:17 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 16:34:17 App: TestKVSwapRebWithTimers got deployed
2021/12/09 16:34:17 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 16:34:17 Starting bucket ops to source bucket
2021/12/09 16:34:18 Removing node(s): [] from the cluster
2021/12/09 16:34:23 Rebalance progress: 12.5
2021/12/09 16:34:28 Rebalance progress: 12.5
2021/12/09 16:34:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:34:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:34:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:34:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 16:34:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:34:39 Rebalance progress: 12.5
2021/12/09 16:34:44 Rebalance progress: 12.5
2021/12/09 16:34:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:34:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:34:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:34:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 16:34:54 Rebalance progress: 12.5
2021/12/09 16:34:59 Rebalance progress: 12.5
2021/12/09 16:35:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:35:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:35:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:35:04 Waiting for src bucket item count to get to: 50000 curr count: 49887
2021/12/09 16:35:09 Waiting for src bucket item count to get to: 50000 curr count: 49887
2021/12/09 16:35:19 Waiting for dst bucket item count to get to: 50000 curr count: 40934
2021/12/09 16:35:24 Waiting for dst bucket item count to get to: 50000 curr count: 40934
2021/12/09 16:35:29 Waiting for dst bucket item count to get to: 50000 curr count: 49418
2021/12/09 16:35:34 Waiting for dst bucket item count to get to: 50000 curr count: 49418
2021/12/09 16:35:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:35:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:35:34 Starting bucket ops to source bucket
2021/12/09 16:36:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:36:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:36:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:36:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:37:03 Waiting for dst bucket item count to get to: 0 curr count: 47145
2021/12/09 16:37:08 Waiting for dst bucket item count to get to: 0 curr count: 47145
2021/12/09 16:37:13 Waiting for dst bucket item count to get to: 0 curr count: 30141
2021/12/09 16:37:18 Waiting for dst bucket item count to get to: 0 curr count: 30141
2021/12/09 16:37:23 Waiting for dst bucket item count to get to: 0 curr count: 15200
2021/12/09 16:37:28 Waiting for dst bucket item count to get to: 0 curr count: 744
2021/12/09 16:37:33 Waiting for dst bucket item count to get to: 0 curr count: 744
2021/12/09 16:37:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:37:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 16:37:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:37:39 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 16:37:44 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 16:37:49 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 16:37:54 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 16:37:59 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 16:38:04 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 16:38:04 App: TestKVSwapRebWithTimers got un-deployed
2021/12/09 16:38:04 No eventing-con process running
2021/12/09 16:38:04 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/09 16:38:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:38:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:38:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:38:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:38:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:38:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (289.53s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/09 16:38:27 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 16:38:27 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/09 16:38:27 No eventing-con process running
2021/12/09 16:38:27 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/09 16:38:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:38:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:38:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:38:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:38:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:38:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:38:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:38:46 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:38:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 16:38:52 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 16:38:57 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 16:39:02 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 16:39:07 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 16:39:07 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/09 16:39:07 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 16:39:07 Starting bucket ops to source bucket
2021/12/09 16:39:07 Removing node(s): [] from the cluster
2021/12/09 16:39:12 Rebalance progress: 12.5
2021/12/09 16:39:17 Rebalance progress: 12.5
2021/12/09 16:39:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:39:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:39:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:39:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:39:27 Rebalance progress: 12.5
2021/12/09 16:39:32 Rebalance progress: 12.5
2021/12/09 16:39:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:39:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:39:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:39:38 Waiting for src bucket item count to get to: 50000 curr count: 27825
2021/12/09 16:39:43 Waiting for src bucket item count to get to: 50000 curr count: 38867
2021/12/09 16:39:48 Waiting for src bucket item count to get to: 50000 curr count: 38867
2021/12/09 16:39:53 Waiting for src bucket item count to get to: 50000 curr count: 49689
2021/12/09 16:39:58 Waiting for src bucket item count to get to: 50000 curr count: 49689
2021/12/09 16:40:08 Waiting for dst bucket item count to get to: 50000 curr count: 39141
2021/12/09 16:40:13 Waiting for dst bucket item count to get to: 50000 curr count: 39141
2021/12/09 16:40:18 Waiting for dst bucket item count to get to: 50000 curr count: 46805
2021/12/09 16:40:23 Waiting for dst bucket item count to get to: 50000 curr count: 46805
2021/12/09 16:40:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:40:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:40:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:40:28 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 16:40:33 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 16:40:38 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 16:40:43 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 16:40:48 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 16:40:53 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 16:40:53 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/09 16:40:53 No eventing-con process running
2021/12/09 16:40:53 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/09 16:40:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:41:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:41:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:41:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:41:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:41:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:41:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:41:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.18s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/09 16:41:21 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 16:41:22 Removing node(s): [] from the cluster
2021/12/09 16:41:27 Rebalance progress: 100
2021/12/09 16:41:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 16:41:32 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/09 16:41:32 No eventing-con process running
2021/12/09 16:41:32 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/09 16:41:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:41:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:41:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:41:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:41:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:41:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:41:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:41:50 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:41:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 16:41:57 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 16:42:02 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 16:42:07 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 16:42:12 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 16:42:12 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/09 16:42:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:42:12 Starting bucket ops to source bucket
2021/12/09 16:42:17 Rebalance progress: 12.5
2021/12/09 16:42:22 Rebalance progress: 12.5
2021/12/09 16:42:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:42:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:42:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:42:27 Waiting for src bucket item count to get to: 50000 curr count: 10053
2021/12/09 16:42:32 Waiting for src bucket item count to get to: 50000 curr count: 24002
2021/12/09 16:42:37 Waiting for src bucket item count to get to: 50000 curr count: 24002
2021/12/09 16:42:42 Waiting for src bucket item count to get to: 50000 curr count: 37298
2021/12/09 16:42:47 Waiting for src bucket item count to get to: 50000 curr count: 37298
2021/12/09 16:42:52 Waiting for src bucket item count to get to: 50000 curr count: 46748
2021/12/09 16:42:57 Waiting for src bucket item count to get to: 50000 curr count: 46748
2021/12/09 16:43:02 Waiting for src bucket item count to get to: 50000 curr count: 46748
2021/12/09 16:43:12 Waiting for dst bucket item count to get to: 50000 curr count: 30856
2021/12/09 16:43:17 Waiting for dst bucket item count to get to: 50000 curr count: 30856
2021/12/09 16:43:22 Waiting for dst bucket item count to get to: 50000 curr count: 44209
2021/12/09 16:43:27 Waiting for dst bucket item count to get to: 50000 curr count: 44209
2021/12/09 16:43:27 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:43:27 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:43:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:43:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 16:43:37 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 16:43:42 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 16:43:47 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 16:43:52 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 16:43:57 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 16:43:57 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/09 16:43:57 No eventing-con process running
2021/12/09 16:43:57 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/09 16:43:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:44:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:44:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:44:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:44:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:44:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:44:21 Waiting for dst bucket item count to get to: 0 curr count: 36660
2021/12/09 16:44:26 Waiting for dst bucket item count to get to: 0 curr count: 36660
2021/12/09 16:44:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.24s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/09 16:44:36 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 16:44:36 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/09 16:44:36 No eventing-con process running
2021/12/09 16:44:36 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/09 16:44:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:44:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:44:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:44:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:44:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:44:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:44:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:44:54 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:44:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 16:45:00 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 16:45:05 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 16:45:10 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 16:45:15 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 16:45:15 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/09 16:45:15 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 16:45:15 Starting bucket ops to source bucket
2021/12/09 16:45:16 Removing node(s): [] from the cluster
2021/12/09 16:45:21 Rebalance progress: 12.5
2021/12/09 16:45:26 Rebalance progress: 12.5
2021/12/09 16:45:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:45:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:45:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:45:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 16:45:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:45:36 Rebalance progress: 12.5
2021/12/09 16:45:41 Rebalance progress: 12.5
2021/12/09 16:45:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:45:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:45:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:45:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 16:45:51 Rebalance progress: 12.5
2021/12/09 16:45:56 Rebalance progress: 12.5
2021/12/09 16:46:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:46:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:46:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:46:06 Waiting for dst bucket item count to get to: 50000 curr count: 27457
2021/12/09 16:46:11 Waiting for dst bucket item count to get to: 50000 curr count: 27457
2021/12/09 16:46:16 Waiting for dst bucket item count to get to: 50000 curr count: 36048
2021/12/09 16:46:21 Waiting for dst bucket item count to get to: 50000 curr count: 36048
2021/12/09 16:46:26 Waiting for dst bucket item count to get to: 50000 curr count: 49038
2021/12/09 16:46:31 Waiting for dst bucket item count to get to: 50000 curr count: 49038
2021/12/09 16:46:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:46:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:46:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:46:36 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 16:46:42 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 16:46:47 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 16:46:52 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 16:46:57 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 16:47:02 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 16:47:02 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/09 16:47:02 No eventing-con process running
2021/12/09 16:47:02 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/09 16:47:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:47:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:47:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:47:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:47:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:47:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:47:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:47:30 Waiting for dst bucket item count to get to: 0 curr count: 18893
2021/12/09 16:47:35 Waiting for dst bucket item count to get to: 0 curr count: 18893
2021/12/09 16:47:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (189.67s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/09 16:47:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 16:47:45 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/09 16:47:45 No eventing-con process running
2021/12/09 16:47:45 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/09 16:47:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:47:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:47:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:47:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:47:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:47:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:48:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:48:04 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:48:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 16:48:10 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 16:48:15 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 16:48:20 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 16:48:25 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 16:48:25 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/09 16:48:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 16:48:25 Starting bucket ops to source bucket
2021/12/09 16:48:25 Removing node(s): [] from the cluster
2021/12/09 16:48:30 Rebalance progress: 12.5
2021/12/09 16:48:35 Rebalance progress: 12.5
2021/12/09 16:48:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:48:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:48:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:48:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:48:45 Rebalance progress: 12.5
2021/12/09 16:48:50 Rebalance progress: 12.5
2021/12/09 16:48:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:48:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:48:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:48:55 Waiting for src bucket item count to get to: 50000 curr count: 32877
2021/12/09 16:49:00 Waiting for src bucket item count to get to: 50000 curr count: 44331
2021/12/09 16:49:05 Waiting for src bucket item count to get to: 50000 curr count: 44331
2021/12/09 16:49:15 Waiting for dst bucket item count to get to: 50000 curr count: 27252
2021/12/09 16:49:20 Waiting for dst bucket item count to get to: 50000 curr count: 27252
2021/12/09 16:49:26 Waiting for dst bucket item count to get to: 50000 curr count: 37052
2021/12/09 16:49:31 Waiting for dst bucket item count to get to: 50000 curr count: 37052
2021/12/09 16:49:36 Waiting for dst bucket item count to get to: 50000 curr count: 37052
2021/12/09 16:49:36 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:49:36 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:49:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:49:41 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 16:49:46 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 16:49:51 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 16:49:56 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 16:50:01 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 16:50:06 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 16:50:06 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/09 16:50:06 No eventing-con process running
2021/12/09 16:50:06 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/09 16:50:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:50:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:50:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:50:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:50:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:50:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:50:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (169.08s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/09 16:50:29 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 16:50:30 Removing node(s): [] from the cluster
2021/12/09 16:50:35 Rebalance progress: 100
2021/12/09 16:50:40 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 16:50:40 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/09 16:50:40 No eventing-con process running
2021/12/09 16:50:40 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/09 16:50:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:50:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:50:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:50:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:50:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:50:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:50:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:50:59 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:51:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 16:51:05 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 16:51:10 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 16:51:15 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 16:51:20 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 16:51:20 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/09 16:51:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:51:20 Starting bucket ops to source bucket
2021/12/09 16:51:25 Rebalance progress: 12.5
2021/12/09 16:51:30 Rebalance progress: 12.5
2021/12/09 16:51:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:51:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:51:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:51:35 Waiting for src bucket item count to get to: 50000 curr count: 12650
2021/12/09 16:51:40 Waiting for src bucket item count to get to: 50000 curr count: 12650
2021/12/09 16:51:45 Waiting for src bucket item count to get to: 50000 curr count: 26368
2021/12/09 16:51:50 Waiting for src bucket item count to get to: 50000 curr count: 38374
2021/12/09 16:51:55 Waiting for src bucket item count to get to: 50000 curr count: 38374
2021/12/09 16:52:00 Waiting for src bucket item count to get to: 50000 curr count: 48585
2021/12/09 16:52:05 Waiting for src bucket item count to get to: 50000 curr count: 48585
2021/12/09 16:52:10 Waiting for src bucket item count to get to: 50000 curr count: 48585
2021/12/09 16:52:20 Waiting for dst bucket item count to get to: 50000 curr count: 33138
2021/12/09 16:52:25 Waiting for dst bucket item count to get to: 50000 curr count: 33138
2021/12/09 16:52:30 Waiting for dst bucket item count to get to: 50000 curr count: 44012
2021/12/09 16:52:35 Waiting for dst bucket item count to get to: 50000 curr count: 44012
2021/12/09 16:52:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:52:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:52:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:52:40 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 16:52:45 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 16:52:50 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 16:52:55 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 16:53:00 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 16:53:05 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 16:53:05 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/09 16:53:05 No eventing-con process running
2021/12/09 16:53:05 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/09 16:53:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:53:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:53:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:53:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:53:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:53:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:53:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:53:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (184.30s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/09 16:53:39 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 16:53:39 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/09 16:53:39 No eventing-con process running
2021/12/09 16:53:39 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/09 16:53:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:53:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:53:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:53:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:53:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:53:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:53:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:53:57 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:53:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 16:54:03 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 16:54:08 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 16:54:13 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 16:54:18 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 16:54:18 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/09 16:54:18 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 16:54:18 Starting bucket ops to source bucket
2021/12/09 16:54:19 Removing node(s): [] from the cluster
2021/12/09 16:54:24 Rebalance progress: 12.5
2021/12/09 16:54:29 Rebalance progress: 12.5
2021/12/09 16:54:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:54:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:54:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:54:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 16:54:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:54:39 Rebalance progress: 12.5
2021/12/09 16:54:44 Rebalance progress: 12.5
2021/12/09 16:54:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:54:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:54:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:54:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 16:54:54 Rebalance progress: 12.5
2021/12/09 16:54:59 Rebalance progress: 12.5
2021/12/09 16:55:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:55:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:55:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:55:04 Waiting for src bucket item count to get to: 50000 curr count: 47921
2021/12/09 16:55:09 Waiting for src bucket item count to get to: 50000 curr count: 47921
2021/12/09 16:55:19 Waiting for dst bucket item count to get to: 50000 curr count: 36917
2021/12/09 16:55:24 Waiting for dst bucket item count to get to: 50000 curr count: 36917
2021/12/09 16:55:29 Waiting for dst bucket item count to get to: 50000 curr count: 43840
2021/12/09 16:55:34 Waiting for dst bucket item count to get to: 50000 curr count: 43840
2021/12/09 16:55:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:55:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:55:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:55:40 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 16:55:45 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 16:55:50 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 16:55:55 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 16:56:00 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 16:56:05 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 16:56:05 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/09 16:56:05 No eventing-con process running
2021/12/09 16:56:05 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/09 16:56:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:56:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:56:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:56:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:56:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:56:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:56:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.66s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/09 16:56:38 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 16:56:38 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/09 16:56:38 No eventing-con process running
2021/12/09 16:56:38 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/09 16:56:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:56:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:56:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:56:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:56:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:56:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:56:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:56:57 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:56:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 16:57:03 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 16:57:08 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 16:57:13 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 16:57:18 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 16:57:18 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/09 16:57:18 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 16:57:18 Starting bucket ops to source bucket
2021/12/09 16:57:18 Removing node(s): [] from the cluster
2021/12/09 16:57:23 Rebalance progress: 12.5
2021/12/09 16:57:28 Rebalance progress: 12.5
2021/12/09 16:57:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:57:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:57:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:57:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 16:57:38 Rebalance progress: 12.5
2021/12/09 16:57:43 Rebalance progress: 12.5
2021/12/09 16:57:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:57:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:57:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 16:57:48 Waiting for src bucket item count to get to: 50000 curr count: 25546
2021/12/09 16:57:53 Waiting for src bucket item count to get to: 50000 curr count: 35363
2021/12/09 16:57:58 Waiting for src bucket item count to get to: 50000 curr count: 35363
2021/12/09 16:58:03 Waiting for src bucket item count to get to: 50000 curr count: 48105
2021/12/09 16:58:08 Waiting for src bucket item count to get to: 50000 curr count: 48105
2021/12/09 16:58:18 Waiting for dst bucket item count to get to: 50000 curr count: 48122
2021/12/09 16:58:24 Waiting for dst bucket item count to get to: 50000 curr count: 48122
2021/12/09 16:58:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 16:58:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 16:58:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 16:58:29 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 16:58:34 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 16:58:39 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 16:58:44 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 16:58:49 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 16:58:54 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 16:58:54 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/09 16:58:54 No eventing-con process running
2021/12/09 16:58:54 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/09 16:58:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:59:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:59:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:59:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:59:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:59:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:59:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 16:59:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.03s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/09 16:59:22 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 16:59:23 Removing node(s): [] from the cluster
2021/12/09 16:59:28 Rebalance progress: 100
2021/12/09 16:59:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 16:59:33 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/09 16:59:33 No eventing-con process running
2021/12/09 16:59:33 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/09 16:59:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:59:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:59:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 16:59:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:59:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:59:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 16:59:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 16:59:51 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 16:59:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 16:59:57 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 17:00:02 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 17:00:07 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 17:00:12 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 17:00:12 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/09 17:00:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:00:12 Starting bucket ops to source bucket
2021/12/09 17:00:17 Rebalance progress: 12.5
2021/12/09 17:00:22 Rebalance progress: 12.5
2021/12/09 17:00:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:00:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:00:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:00:27 Waiting for src bucket item count to get to: 50000 curr count: 8793
2021/12/09 17:00:32 Waiting for src bucket item count to get to: 50000 curr count: 20595
2021/12/09 17:00:37 Waiting for src bucket item count to get to: 50000 curr count: 20595
2021/12/09 17:00:43 Waiting for src bucket item count to get to: 50000 curr count: 32320
2021/12/09 17:00:48 Waiting for src bucket item count to get to: 50000 curr count: 32320
2021/12/09 17:00:53 Waiting for src bucket item count to get to: 50000 curr count: 42629
2021/12/09 17:00:58 Waiting for src bucket item count to get to: 50000 curr count: 42629
2021/12/09 17:01:03 Waiting for src bucket item count to get to: 50000 curr count: 42629
2021/12/09 17:01:13 Waiting for dst bucket item count to get to: 50000 curr count: 40695
2021/12/09 17:01:18 Waiting for dst bucket item count to get to: 50000 curr count: 40695
2021/12/09 17:01:18 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 17:01:18 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 17:01:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:01:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 17:01:28 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 17:01:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 17:01:38 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 17:01:43 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 17:01:48 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 17:01:48 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/09 17:01:48 No eventing-con process running
2021/12/09 17:01:48 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/09 17:01:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:01:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:01:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:01:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:02:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:02:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 17:02:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (169.08s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/09 17:02:16 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 17:02:16 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/09 17:02:16 No eventing-con process running
2021/12/09 17:02:17 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/09 17:02:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:02:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:02:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:02:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:02:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:02:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:02:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:02:35 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:02:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 17:02:41 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 17:02:46 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 17:02:51 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 17:02:56 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 17:02:56 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/09 17:02:56 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 17:02:56 Starting bucket ops to source bucket
2021/12/09 17:02:57 Removing node(s): [] from the cluster
2021/12/09 17:03:02 Rebalance progress: 12.5
2021/12/09 17:03:07 Rebalance progress: 12.5
2021/12/09 17:03:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:03:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:03:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:03:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 17:03:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:03:17 Rebalance progress: 12.5
2021/12/09 17:03:22 Rebalance progress: 12.5
2021/12/09 17:03:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:03:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:03:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:03:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 17:03:32 Rebalance progress: 12.5
2021/12/09 17:03:37 Rebalance progress: 12.5
2021/12/09 17:03:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:03:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:03:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:03:42 Waiting for src bucket item count to get to: 50000 curr count: 48969
2021/12/09 17:03:47 Waiting for src bucket item count to get to: 50000 curr count: 48969
2021/12/09 17:03:57 Waiting for dst bucket item count to get to: 50000 curr count: 44676
2021/12/09 17:04:02 Waiting for dst bucket item count to get to: 50000 curr count: 44676
2021/12/09 17:04:02 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 17:04:02 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 17:04:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:04:07 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 17:04:13 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 17:04:18 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 17:04:23 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 17:04:28 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 17:04:28 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/09 17:04:28 No eventing-con process running
2021/12/09 17:04:28 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/09 17:04:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:04:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:04:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:04:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:04:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:04:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 17:04:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (159.84s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/09 17:04:56 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 17:04:56 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/09 17:04:56 No eventing-con process running
2021/12/09 17:04:56 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/09 17:04:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:05:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:05:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:05:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:05:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:05:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:05:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:05:15 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:05:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 17:05:21 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 17:05:26 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 17:05:31 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 17:05:36 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 17:05:36 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/09 17:05:36 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:05:36 Starting bucket ops to source bucket
2021/12/09 17:05:37 Removing node(s): [] from the cluster
2021/12/09 17:05:42 Rebalance progress: 12.5
2021/12/09 17:05:47 Rebalance progress: 12.5
2021/12/09 17:05:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:05:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:05:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:05:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:05:57 Rebalance progress: 12.5
2021/12/09 17:06:02 Rebalance progress: 12.5
2021/12/09 17:06:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:06:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:06:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:06:07 Waiting for src bucket item count to get to: 50000 curr count: 27189
2021/12/09 17:06:12 Waiting for src bucket item count to get to: 50000 curr count: 38404
2021/12/09 17:06:17 Waiting for src bucket item count to get to: 50000 curr count: 38404
2021/12/09 17:06:22 Waiting for src bucket item count to get to: 50000 curr count: 49387
2021/12/09 17:06:27 Waiting for src bucket item count to get to: 50000 curr count: 49387
2021/12/09 17:06:37 Waiting for dst bucket item count to get to: 50000 curr count: 46733
2021/12/09 17:06:42 Waiting for dst bucket item count to get to: 50000 curr count: 46733
2021/12/09 17:06:42 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 17:06:42 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 17:06:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:06:47 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 17:06:52 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 17:06:57 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 17:07:02 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 17:07:07 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 17:07:12 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 17:07:12 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/09 17:07:12 No eventing-con process running
2021/12/09 17:07:12 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/09 17:07:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:07:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:07:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 17:07:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 17:07:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:07:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:07:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:07:41 Waiting for dst bucket item count to get to: 0 curr count: 36461
2021/12/09 17:07:46 Waiting for dst bucket item count to get to: 0 curr count: 36461
2021/12/09 17:07:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (179.36s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/09 17:07:51 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:07:51 Removing node(s): [] from the cluster
2021/12/09 17:07:56 Rebalance progress: 100
2021/12/09 17:08:01 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 17:08:01 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/09 17:08:01 No eventing-con process running
2021/12/09 17:08:01 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/09 17:08:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:08:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:08:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:08:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:08:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:08:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:08:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:08:20 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:08:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 17:08:26 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 17:08:31 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 17:08:36 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 17:08:41 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 17:08:41 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/09 17:08:41 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:08:41 Starting bucket ops to source bucket
2021/12/09 17:08:46 Rebalance progress: 12.5
2021/12/09 17:08:51 Rebalance progress: 12.5
2021/12/09 17:08:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:08:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:08:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:08:56 Waiting for src bucket item count to get to: 50000 curr count: 11535
2021/12/09 17:09:01 Waiting for src bucket item count to get to: 50000 curr count: 25000
2021/12/09 17:09:06 Waiting for src bucket item count to get to: 50000 curr count: 25000
2021/12/09 17:09:11 Waiting for src bucket item count to get to: 50000 curr count: 35003
2021/12/09 17:09:16 Waiting for src bucket item count to get to: 50000 curr count: 35003
2021/12/09 17:09:21 Waiting for src bucket item count to get to: 50000 curr count: 46569
2021/12/09 17:09:26 Waiting for src bucket item count to get to: 50000 curr count: 46569
2021/12/09 17:09:31 Waiting for src bucket item count to get to: 50000 curr count: 46569
2021/12/09 17:09:41 Waiting for dst bucket item count to get to: 50000 curr count: 37673
2021/12/09 17:09:46 Waiting for dst bucket item count to get to: 50000 curr count: 37673
2021/12/09 17:09:46 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 17:09:46 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 17:09:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:09:51 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 17:09:56 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 17:10:01 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 17:10:06 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 17:10:11 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 17:10:11 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/09 17:10:11 No eventing-con process running
2021/12/09 17:10:11 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/09 17:10:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:10:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:10:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 17:10:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 17:10:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:10:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:10:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:10:40 Waiting for dst bucket item count to get to: 0 curr count: 2428
2021/12/09 17:10:45 Waiting for dst bucket item count to get to: 0 curr count: 2428
2021/12/09 17:10:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.26s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/09 17:10:55 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 17:10:55 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/09 17:10:55 No eventing-con process running
2021/12/09 17:10:55 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/09 17:10:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:11:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:11:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:11:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:11:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:11:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:11:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:11:13 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:11:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 17:11:20 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 17:11:25 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 17:11:30 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 17:11:35 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 17:11:35 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/09 17:11:35 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:11:35 Starting bucket ops to source bucket
2021/12/09 17:11:35 Removing node(s): [] from the cluster
2021/12/09 17:11:40 Rebalance progress: 12.5
2021/12/09 17:11:45 Rebalance progress: 12.5
2021/12/09 17:11:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:11:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:11:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:11:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 17:11:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:11:55 Rebalance progress: 12.5
2021/12/09 17:12:00 Rebalance progress: 12.5
2021/12/09 17:12:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:12:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 17:12:11 Rebalance progress: 12.5
2021/12/09 17:12:16 Rebalance progress: 12.5
2021/12/09 17:12:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:12:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:12:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:12:26 Waiting for dst bucket item count to get to: 50000 curr count: 36790
2021/12/09 17:12:31 Waiting for dst bucket item count to get to: 50000 curr count: 36790
2021/12/09 17:12:36 Waiting for dst bucket item count to get to: 50000 curr count: 44268
2021/12/09 17:12:41 Waiting for dst bucket item count to get to: 50000 curr count: 44268
2021/12/09 17:12:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 17:12:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 17:12:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:12:46 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 17:12:51 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 17:12:56 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 17:13:01 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 17:13:06 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 17:13:11 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 17:13:11 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/09 17:13:11 No eventing-con process running
2021/12/09 17:13:11 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/09 17:13:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:13:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:13:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 17:13:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 17:13:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:13:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:13:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:13:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (169.57s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/09 17:13:45 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:13:45 App: bucket_op_on_update got un-deployed
2021/12/09 17:13:45 No eventing-con process running
2021/12/09 17:13:45 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 17:13:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:13:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:13:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:13:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:13:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:13:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:14:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:14:03 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:14:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 17:14:09 Starting bucket ops to source bucket
2021/12/09 17:14:14 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:14:19 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:14:24 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:14:24 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:14:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:14:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:14:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:14:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:14:30 Rebalance progress: 12.5
2021/12/09 17:14:35 Rebalance progress: 12.5
2021/12/09 17:14:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:14:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:14:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:14:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 17:14:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:14:45 Rebalance progress: 12.5
2021/12/09 17:14:50 Rebalance progress: 12.5
2021/12/09 17:14:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:14:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:14:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:14:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 17:14:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:15:01 Rebalance progress: 12.5
2021/12/09 17:15:06 Rebalance progress: 12.5
2021/12/09 17:15:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:15:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:15:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:15:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:15:16 Rebalance progress: 12.5
2021/12/09 17:15:21 Rebalance progress: 12.5
2021/12/09 17:15:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:15:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:15:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:15:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 17:15:31 Rebalance progress: 12.5
2021/12/09 17:15:36 Rebalance progress: 12.5
2021/12/09 17:15:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:15:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:15:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:15:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 17:15:46 Rebalance progress: 12.5
2021/12/09 17:15:51 Rebalance progress: 12.5
2021/12/09 17:15:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:15:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:15:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:15:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:16:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:16:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:16:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:16:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:16:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:16:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:16:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:16:26 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:16:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 17:16:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:16:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:16:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10656
2021/12/09 17:16:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:16:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:16:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:16:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (190.29s)
=== RUN   TestKVForceFailover
2021/12/09 17:17:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:17:00 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:17:00 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:17:00 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 17:17:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:17:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:17:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:17:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:17:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:17:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:17:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:17:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:17:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:17:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:17:19 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:17:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 17:17:20 Starting bucket ops to source bucket
2021/12/09 17:17:25 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:17:30 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:17:35 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:17:40 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:17:40 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:17:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:17:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:17:40 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/09 17:18:00 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/09 17:18:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 17:18:05 Rebalance progress: 12.5
2021/12/09 17:18:10 Rebalance progress: 12.5
2021/12/09 17:18:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:18:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:18:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:18:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/09 17:18:20 Rebalance progress: 12.5
2021/12/09 17:18:25 Rebalance progress: 12.5
2021/12/09 17:18:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:18:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:18:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:18:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:18:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:18:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:18:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:18:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:18:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:19:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:19:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:19:01 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:19:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 17:19:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:19:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:19:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7061
2021/12/09 17:19:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7061
2021/12/09 17:19:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:19:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:19:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:19:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.58s)
=== RUN   TestKVHardFailoverRecovery
2021/12/09 17:19:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:19:34 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:19:34 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:19:34 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 17:19:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:19:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:19:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:19:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:19:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:19:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:19:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:19:53 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:19:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 17:19:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 17:20:04 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:20:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:20:14 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:20:14 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:20:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:20:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:20:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:20:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:20:20 Rebalance progress: 12.5
2021/12/09 17:20:25 Rebalance progress: 12.5
2021/12/09 17:20:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:20:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:20:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:20:30 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/09 17:21:30 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/09 17:21:35 Rebalance progress: 12.5
2021/12/09 17:21:40 Rebalance progress: 12.5
2021/12/09 17:21:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:21:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:21:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:21:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:21:50 Rebalance progress: 12.5
2021/12/09 17:21:55 Rebalance progress: 12.5
2021/12/09 17:22:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:22:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:22:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:22:00 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:22:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:22:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:22:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:22:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:22:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:22:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:22:30 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:22:30 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:22:30 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 17:22:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:22:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:22:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12064
2021/12/09 17:22:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12064
2021/12/09 17:22:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:22:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:22:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:22:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12064
2021/12/09 17:22:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (214.42s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/09 17:23:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:23:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:23:09 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:23:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 17:23:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:23:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:23:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:23:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:23:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:23:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:23:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:23:27 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:23:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 17:23:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 17:23:38 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:23:43 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:23:48 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:23:48 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:23:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:23:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:23:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:23:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 17:23:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 17:23:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:23:54 Rebalance progress: 12.5
2021/12/09 17:23:59 Rebalance progress: 12.5
2021/12/09 17:24:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:24:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:24:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:24:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 17:24:09 Rebalance progress: 12.5
2021/12/09 17:24:14 Rebalance progress: 12.5
2021/12/09 17:24:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:24:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:24:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:24:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:24:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:24:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:24:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:24:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:24:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:24:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:24:45 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:24:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 17:24:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:24:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:24:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4592
2021/12/09 17:24:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:24:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:25:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:25:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4592
2021/12/09 17:25:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (129.57s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/09 17:25:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:25:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:25:18 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:25:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 17:25:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:25:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:25:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:25:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:25:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:25:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:25:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:25:32 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:25:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 17:25:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 17:25:43 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:25:48 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:25:53 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:25:53 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:25:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:25:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:25:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:25:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:25:58 Rebalance progress: 12.5
2021/12/09 17:26:03 Rebalance progress: 12.5
2021/12/09 17:26:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:26:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:26:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:26:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 17:26:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:26:14 Rebalance progress: 12.5
2021/12/09 17:26:19 Rebalance progress: 12.5
2021/12/09 17:26:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:26:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:26:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:26:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 17:26:29 Rebalance progress: 12.5
2021/12/09 17:26:34 Rebalance progress: 12.5
2021/12/09 17:26:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:26:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:26:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:26:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:26:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:26:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:26:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:26:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:27:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:27:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:27:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:27:09 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:27:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 17:27:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:27:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:27:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6081
2021/12/09 17:27:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6081
2021/12/09 17:27:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:27:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:27:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:27:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6081
2021/12/09 17:27:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:27:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:27:43 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:27:43 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:27:43 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 17:27:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:27:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:27:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:27:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:27:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:27:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.18s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/09 17:28:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:28:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:28:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:28:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 17:28:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:28:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:28:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:28:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:28:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:28:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:28:25 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:28:25 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 17:28:25 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:28:25 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/09 17:28:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:28:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:28:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:28:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:28:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:28:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:28:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:28:39 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:28:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:28:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:28:40 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:28:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/09 17:28:46 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 17:28:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:28:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:29:01 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:29:01 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/09 17:29:06 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 17:29:11 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 17:29:16 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 17:29:16 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:29:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:29:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:29:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:29:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:29:22 Rebalance progress: 12.5
2021/12/09 17:29:27 Rebalance progress: 12.5
2021/12/09 17:29:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:29:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:29:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:29:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 17:29:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:29:37 Rebalance progress: 12.5
2021/12/09 17:29:42 Rebalance progress: 12.5
2021/12/09 17:29:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:29:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:29:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:29:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 17:29:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:29:52 Rebalance progress: 12.5
2021/12/09 17:29:57 Rebalance progress: 12.5
2021/12/09 17:30:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:30:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:30:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:30:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:30:08 Rebalance progress: 12.5
2021/12/09 17:30:13 Rebalance progress: 12.5
2021/12/09 17:30:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:30:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:30:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:30:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 17:30:23 Rebalance progress: 12.5
2021/12/09 17:30:28 Rebalance progress: 12.5
2021/12/09 17:30:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:30:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:30:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:30:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 17:30:38 Rebalance progress: 12.5
2021/12/09 17:30:43 Rebalance progress: 12.5
2021/12/09 17:30:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:30:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:30:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:30:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:30:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:30:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:31:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:31:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:31:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:31:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:31:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:31:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:31:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:31:28 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:31:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 17:31:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:31:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:31:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7793
2021/12/09 17:31:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7793
2021/12/09 17:31:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:31:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:31:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:31:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:31:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:32:00 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:32:05 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:32:10 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:32:15 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:32:20 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:32:25 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:32:25 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 17:32:25 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:32:25 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/09 17:32:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:32:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:32:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:32:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:32:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:32:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (281.99s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/09 17:32:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:32:49 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:32:49 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:32:49 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 17:32:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:32:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:32:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:32:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:33:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:33:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:33:07 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:33:07 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 17:33:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:33:07 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/09 17:33:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:33:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:33:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:33:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:33:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:33:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:33:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:33:21 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:33:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:33:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:33:22 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:33:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/09 17:33:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 17:33:33 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:33:38 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:33:43 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:33:43 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/09 17:33:48 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 17:33:53 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 17:33:58 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 17:33:58 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:33:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:33:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:33:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:33:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:34:04 Rebalance progress: 12.5
2021/12/09 17:34:09 Rebalance progress: 12.5
2021/12/09 17:34:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:34:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:34:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:34:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 17:34:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:34:19 Rebalance progress: 12.5
2021/12/09 17:34:24 Rebalance progress: 12.5
2021/12/09 17:34:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:34:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:34:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:34:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 17:34:35 Rebalance progress: 12.5
2021/12/09 17:34:40 Rebalance progress: 12.5
2021/12/09 17:34:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:34:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:34:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:34:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:34:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:34:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:35:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:35:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:35:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:35:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:35:15 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:35:15 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:35:15 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 17:35:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:35:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:35:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7655
2021/12/09 17:35:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:35:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:35:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:35:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7655
2021/12/09 17:35:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7655
2021/12/09 17:35:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:35:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:35:51 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:35:56 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:36:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:36:06 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:36:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:36:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 17:36:16 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 17:36:16 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:36:16 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/09 17:36:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:36:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:36:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:36:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:36:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:36:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (231.40s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/09 17:36:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:36:40 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:36:40 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:36:40 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 17:36:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:36:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:36:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:36:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:36:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:36:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:36:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:36:59 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:37:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 17:37:05 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 17:37:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:37:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:37:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:37:20 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:37:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:37:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:37:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:37:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:37:40 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:37:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:37:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:37:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:37:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:37:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:37:56 Rebalance progress: 12.5
2021/12/09 17:38:01 Rebalance progress: 12.5
2021/12/09 17:38:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:38:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:38:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:38:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 17:38:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:38:11 Rebalance progress: 12.5
2021/12/09 17:38:16 Rebalance progress: 12.5
2021/12/09 17:38:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:38:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 17:38:26 Rebalance progress: 12.5
2021/12/09 17:38:31 Rebalance progress: 12.5
2021/12/09 17:38:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:38:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:38:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:38:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:38:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:38:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:38:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:38:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:39:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:39:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:39:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:39:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:39:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 17:39:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:39:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:39:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9141
2021/12/09 17:39:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:39:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:39:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:39:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2678
2021/12/09 17:39:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2678
2021/12/09 17:39:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.24s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/09 17:39:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:39:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:39:45 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:39:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 17:39:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:39:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:39:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:39:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:39:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:39:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:40:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:40:04 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:40:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 17:40:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 17:40:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:40:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:40:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:40:25 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:40:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:40:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:40:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:40:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:40:30 Rebalance progress: 12.5
2021/12/09 17:40:35 Rebalance progress: 12.5
2021/12/09 17:40:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:40:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:40:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:40:40 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:40:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:40:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:40:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:40:55 Rebalance progress: 12.5
2021/12/09 17:41:00 Rebalance progress: 12.5
2021/12/09 17:41:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:41:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:41:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:41:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 17:41:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 17:41:11 Rebalance progress: 12.5
2021/12/09 17:41:16 Rebalance progress: 12.5
2021/12/09 17:41:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:41:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:41:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:41:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 17:41:26 Rebalance progress: 12.5
2021/12/09 17:41:31 Rebalance progress: 12.5
2021/12/09 17:41:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:41:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:41:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:41:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:41:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:41:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:41:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:41:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:42:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:42:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:42:06 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:42:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:42:06 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 17:42:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:42:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:42:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8589
2021/12/09 17:42:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:42:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:42:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:42:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (169.54s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/09 17:42:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 17:42:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 17:42:35 Rebalance progress: 100
2021/12/09 17:42:35 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:43:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:43:05 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 17:43:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 17:43:11 Removing node(s): [] from the cluster
2021/12/09 17:43:11 Starting bucket ops to source bucket
2021/12/09 17:43:16 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/09 17:43:21 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:43:26 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 17:43:26 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:43:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:43:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 17:43:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 17:43:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:43:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:43:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:43:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:43:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:43:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 17:43:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 17:43:57 eventingSetup.go:46: No eventing-con process running
2021/12/09 17:43:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 17:43:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:44:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:44:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1504
2021/12/09 17:44:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 17:44:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 17:44:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 17:44:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 716
2021/12/09 17:44:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 716
2021/12/09 17:44:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.67s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12012.653s

Rebalance tests took 3 hours, 20 minutes and 15 seconds.

Testing Succeeded


Version: versions-09.12.2021-10.07.cfg
Build Log: make-09.12.2021-10.07.log
Server Log: logs-09.12.2021-10.07.tar.gz

Finished