Building

Started building at 2021/12/04 03:23:28
Calculating base
Updating mirror
Basing run on 7.1.0-1838 2229b57ce
Updating tree for run 04.12.2021-03.23
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at a540074, changes since last good build: 
 a5400747 Merge remote-tracking branch 'couchbase/unstable' into HEAD
 bda7dff9 MB-49873 use correct interface conversion for MsgStreamError
 7e5f45e6 MB-100 Use %s formatter for logging rebalance token
 442d4ecd MB-48243 Cleanup erroneous MoveIndexToken in Janitor
 53a57765 MB-48243 Cleanup erroneous tokens as a part of orphan token cleanup
 ac596d83 MB-39662 enable OSO by default
 0088b4f8 MB-43796 use true_with_seqno_advanced for OSO mode
 739da27c MB-43796 enable OSO after cluster upgrade to 7.1
 badb98da MB-49851: Revert "MB-48897: Enable plasma in-mem compression configs"
 71e29187 MB-46068 Re-Enable timestamp validation for partition merge
kv_engine is at 7e709d9, changes since last good build: none
ns_server is at 207e6c6, changes since last good build: 
 207e6c644 MB-49830: Add event log when 'nodedown' is rcvd in ns_node_disco
 7e8b35a11 MB-49765 Remove use of assert_is_65
 631182661 MB-49765 Remove use of is_version_65
 001be57c4 MB-49765 Remove use of is_cluster_65
 dbe669c52 MB-49765 Remove VERSION_60 macro
 f45baf2db MB-49765 Remove support for upgrading to 6.5
 c4ca754a8 MB-49765 Make 6.5 the minimum supported version
 5e7c9e5f6 Revert "MB-49871: Rename kv data_size limit to data_size_mib"
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at c1989af, changes since last good build: none
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 7060ef8, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable d19f8a9e] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable d5f7cc94] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 4b6ff9ff] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable e17c7001] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 89597768] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable 355202ab] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/04 03:39:51
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/04 03:41:34 Started functional
2021/12/04 03:41:45 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/04 03:41:57 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/04 03:41:57 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 03:42:02 Removing node(s): [] from the cluster
2021/12/04 03:42:07 Rebalance progress: 2.785011574074074
2021/12/04 03:42:12 Rebalance progress: 7.360387731481481
2021/12/04 03:42:17 Rebalance progress: 10.59751157407407
2021/12/04 03:42:22 Rebalance progress: 15.2271412037037
2021/12/04 03:42:28 Rebalance progress: 18.51851851851852
2021/12/04 03:42:32 Rebalance progress: 23.14814814814815
2021/12/04 03:42:37 Rebalance progress: 27.77777777777778
2021/12/04 03:42:42 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/04 03:42:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:43:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:43:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:43:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:43:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:44:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:44:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:44:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:44:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:45:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:45:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:45:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:45:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 03:46:03 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:46:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/04 03:46:09 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/04 03:46:14 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/04 03:46:19 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/04 03:46:24 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/04 03:46:24 App: TestAdvancedGetOps got deployed
2021/12/04 03:46:24 Starting bucket ops to source bucket
2021/12/04 03:46:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 03:46:35 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 03:46:35 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 03:46:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:46:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:46:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:46:40 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/04 03:46:45 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/04 03:46:50 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/04 03:46:55 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/04 03:46:55 App: TestAdvancedGetOps got un-deployed
2021/12/04 03:46:55 No eventing-con process running
2021/12/04 03:46:55 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/04 03:46:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:47:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 03:47:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:47:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:47:18 Waiting for dst bucket item count to get to: 0 curr count: 81
2021/12/04 03:47:23 Waiting for dst bucket item count to get to: 0 curr count: 31
2021/12/04 03:47:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (275.26s)
=== RUN   TestAdvancedInsertOps
2021/12/04 03:47:23 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:47:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/04 03:47:30 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 03:47:35 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 03:47:40 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 03:47:45 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 03:47:45 App: TestAdvancedInsertOps got deployed
2021/12/04 03:47:45 Starting bucket ops to source bucket
2021/12/04 03:47:51 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/04 03:47:56 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/04 03:47:56 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/04 03:47:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:47:56 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:47:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:48:01 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 03:48:06 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 03:48:11 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 03:48:16 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 03:48:16 App: TestAdvancedInsertOps got un-deployed
2021/12/04 03:48:16 No eventing-con process running
2021/12/04 03:48:16 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/04 03:48:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:48:28 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/04 03:48:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:48:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:48:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 03:48:44 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/04 03:48:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:48:44 Testing insert operation on source bucket
2021/12/04 03:48:44 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:48:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/04 03:48:51 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 03:48:56 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 03:49:01 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 03:49:06 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 03:49:06 App: TestAdvancedInsertOps got deployed
2021/12/04 03:49:06 Starting bucket ops to source bucket
2021/12/04 03:49:12 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/04 03:49:12 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/04 03:49:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:49:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:49:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:49:17 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 03:49:22 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 03:49:27 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 03:49:32 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 03:49:32 App: TestAdvancedInsertOps got un-deployed
2021/12/04 03:49:32 No eventing-con process running
2021/12/04 03:49:32 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/04 03:49:33 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:49:43 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/04 03:49:48 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 03:49:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:49:50 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:49:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (151.82s)
=== RUN   TestAdvancedUpsertOps
2021/12/04 03:49:55 Starting bucket ops to source bucket
2021/12/04 03:49:56 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:49:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/04 03:50:02 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 03:50:07 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 03:50:12 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 03:50:17 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 03:50:17 App: TestAdvancedUpsertOps got deployed
2021/12/04 03:50:17 Starting bucket ops to source bucket
2021/12/04 03:50:23 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 03:50:28 Waiting for dst bucket item count to get to: 0 curr count: 493
2021/12/04 03:50:33 Waiting for dst bucket item count to get to: 0 curr count: 493
2021/12/04 03:50:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:50:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:50:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:50:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:50:38 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 03:50:43 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 03:50:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 03:50:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 03:50:53 App: TestAdvancedUpsertOps got un-deployed
2021/12/04 03:50:53 No eventing-con process running
2021/12/04 03:50:53 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/04 03:50:55 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:51:05 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 03:51:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:51:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:51:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:51:12 Testing upsert operation on source bucket
2021/12/04 03:51:12 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:51:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/04 03:51:18 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 03:51:23 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 03:51:28 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 03:51:33 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 03:51:33 App: TestAdvancedUpsertOps got deployed
2021/12/04 03:51:33 Starting bucket ops to source bucket
2021/12/04 03:51:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:51:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:51:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:51:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:51:39 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 03:51:44 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 03:51:49 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 03:51:54 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 03:51:54 App: TestAdvancedUpsertOps got un-deployed
2021/12/04 03:51:54 No eventing-con process running
2021/12/04 03:51:54 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/04 03:51:56 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:52:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:52:04 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:52:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (133.24s)
=== RUN   TestAdvancedReplaceOps
2021/12/04 03:52:09 Starting bucket ops to source bucket
2021/12/04 03:52:09 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:52:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/04 03:52:16 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 03:52:21 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 03:52:26 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 03:52:31 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 03:52:31 App: TestAdvancedReplaceOps got deployed
2021/12/04 03:52:31 Starting bucket ops to source bucket
2021/12/04 03:52:37 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 03:52:42 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 03:52:47 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/04 03:52:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:52:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:52:47 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:52:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:52:52 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 03:52:57 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 03:53:02 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 03:53:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 03:53:07 App: TestAdvancedReplaceOps got un-deployed
2021/12/04 03:53:07 No eventing-con process running
2021/12/04 03:53:07 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/04 03:53:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:53:19 Waiting for dst bucket item count to get to: 0 curr count: 969
2021/12/04 03:53:24 Waiting for dst bucket item count to get to: 0 curr count: 457
2021/12/04 03:53:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:53:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:53:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:53:31 Testing upsert operation on source bucket
2021/12/04 03:53:31 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:53:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/04 03:53:37 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 03:53:42 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 03:53:47 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 03:53:52 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 03:53:52 App: TestAdvancedReplaceOps got deployed
2021/12/04 03:53:52 Starting bucket ops to source bucket
2021/12/04 03:53:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:53:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:53:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:53:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:53:58 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 03:54:03 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 03:54:08 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 03:54:13 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 03:54:13 App: TestAdvancedReplaceOps got un-deployed
2021/12/04 03:54:13 No eventing-con process running
2021/12/04 03:54:13 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/04 03:54:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:54:25 Waiting for dst bucket item count to get to: 0 curr count: 357
2021/12/04 03:54:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:54:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:54:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (142.92s)
=== RUN   TestAdvancedDeleteOps
2021/12/04 03:54:31 Starting bucket ops to source bucket
2021/12/04 03:54:32 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:54:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/04 03:54:38 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 03:54:43 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 03:54:48 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 03:54:53 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 03:54:53 App: TestAdvancedDeleteOps got deployed
2021/12/04 03:54:53 Starting bucket ops to source bucket
2021/12/04 03:54:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 03:55:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 03:55:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:55:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:55:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:55:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:55:08 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 03:55:13 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 03:55:18 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 03:55:23 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 03:55:23 App: TestAdvancedDeleteOps got un-deployed
2021/12/04 03:55:23 No eventing-con process running
2021/12/04 03:55:23 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/04 03:55:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:55:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 03:55:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:55:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:55:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:55:42 Testing delete operation on source bucket
2021/12/04 03:55:42 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:55:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/04 03:55:48 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 03:55:53 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 03:55:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 03:56:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 03:56:03 App: TestAdvancedDeleteOps got deployed
2021/12/04 03:56:03 Starting bucket ops to source bucket
2021/12/04 03:56:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:56:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:56:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:56:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:56:08 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 03:56:13 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 03:56:18 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 03:56:23 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 03:56:23 App: TestAdvancedDeleteOps got un-deployed
2021/12/04 03:56:23 No eventing-con process running
2021/12/04 03:56:23 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/04 03:56:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:56:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:56:32 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:56:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.69s)
=== RUN   TestEnoentAdvancedGet
2021/12/04 03:56:37 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:56:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/04 03:56:43 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 03:56:48 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 03:56:53 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 03:56:58 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 03:56:58 App: TestEnoentAdvancedGet got deployed
2021/12/04 03:56:58 Starting bucket ops to source bucket
2021/12/04 03:57:04 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 03:57:09 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/04 03:57:09 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 03:57:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:57:09 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:57:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:57:14 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 03:57:19 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 03:57:24 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 03:57:29 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 03:57:29 App: TestEnoentAdvancedGet got un-deployed
2021/12/04 03:57:29 No eventing-con process running
2021/12/04 03:57:29 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/04 03:57:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:57:40 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/04 03:57:45 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/04 03:57:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:57:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:57:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 03:57:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:57:57 Testing get enoent operation on source bucket
2021/12/04 03:57:57 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:57:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/04 03:58:03 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 03:58:08 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 03:58:13 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 03:58:18 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 03:58:18 App: TestEnoentAdvancedGet got deployed
2021/12/04 03:58:18 Starting bucket ops to source bucket
2021/12/04 03:58:24 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/04 03:58:29 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/04 03:58:29 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/04 03:58:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:58:29 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:58:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:58:34 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 03:58:39 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 03:58:44 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 03:58:49 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 03:58:49 App: TestEnoentAdvancedGet got un-deployed
2021/12/04 03:58:49 No eventing-con process running
2021/12/04 03:58:49 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/04 03:58:50 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:59:00 Waiting for dst bucket item count to get to: 0 curr count: 164
2021/12/04 03:59:05 Waiting for dst bucket item count to get to: 0 curr count: 164
2021/12/04 03:59:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 03:59:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 03:59:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.22s)
=== RUN   TestEnoentAdvancedReplace
2021/12/04 03:59:12 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 03:59:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 03:59:18 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 03:59:23 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 03:59:28 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 03:59:33 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 03:59:33 App: TestEnoentAdvancedReplace got deployed
2021/12/04 03:59:33 Starting bucket ops to source bucket
2021/12/04 03:59:39 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 03:59:44 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/04 03:59:44 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 03:59:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 03:59:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 03:59:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 03:59:49 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 03:59:54 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 03:59:59 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 04:00:04 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 04:00:04 App: TestEnoentAdvancedReplace got un-deployed
2021/12/04 04:00:04 No eventing-con process running
2021/12/04 04:00:04 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/04 04:00:06 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:00:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:00:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:00:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:00:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:00:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:00:27 Testing get enoent operation on source bucket
2021/12/04 04:00:27 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:00:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 04:00:34 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 04:00:39 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 04:00:44 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 04:00:49 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 04:00:49 App: TestEnoentAdvancedReplace got deployed
2021/12/04 04:00:49 Starting bucket ops to source bucket
2021/12/04 04:00:54 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/04 04:00:59 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/04 04:00:59 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/04 04:00:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:00:59 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:00:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:01:04 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 04:01:09 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 04:01:14 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 04:01:19 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 04:01:19 App: TestEnoentAdvancedReplace got un-deployed
2021/12/04 04:01:19 No eventing-con process running
2021/12/04 04:01:19 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/04 04:01:21 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:01:31 Waiting for dst bucket item count to get to: 0 curr count: 186
2021/12/04 04:01:36 Waiting for dst bucket item count to get to: 0 curr count: 186
2021/12/04 04:01:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:01:38 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:01:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.20s)
=== RUN   TestEnoentAdvancedDelete
2021/12/04 04:01:43 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:01:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 04:01:49 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 04:01:54 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 04:01:59 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 04:02:04 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 04:02:04 App: TestEnoentAdvancedDelete got deployed
2021/12/04 04:02:04 Starting bucket ops to source bucket
2021/12/04 04:02:09 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 04:02:14 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/04 04:02:14 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 04:02:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:02:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:02:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:02:19 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 04:02:24 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 04:02:29 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 04:02:34 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 04:02:34 App: TestEnoentAdvancedDelete got un-deployed
2021/12/04 04:02:34 No eventing-con process running
2021/12/04 04:02:34 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/04 04:02:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:02:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:02:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:02:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:02:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:02:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:02:58 Testing get enoent operation on source bucket
2021/12/04 04:02:58 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:02:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 04:03:04 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 04:03:09 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 04:03:14 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 04:03:19 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 04:03:19 App: TestEnoentAdvancedDelete got deployed
2021/12/04 04:03:19 Starting bucket ops to source bucket
2021/12/04 04:03:24 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/04 04:03:29 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/04 04:03:29 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/04 04:03:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:03:29 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:03:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:03:34 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 04:03:39 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 04:03:44 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 04:03:49 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 04:03:49 App: TestEnoentAdvancedDelete got un-deployed
2021/12/04 04:03:49 No eventing-con process running
2021/12/04 04:03:49 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/04 04:03:51 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:04:01 Waiting for dst bucket item count to get to: 0 curr count: 188
2021/12/04 04:04:06 Waiting for dst bucket item count to get to: 0 curr count: 188
2021/12/04 04:04:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:04:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:04:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.06s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/04 04:04:13 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:04:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/04 04:04:19 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 04:04:24 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 04:04:29 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 04:04:34 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 04:04:34 App: TestAdvancedInsertKeyExist got deployed
2021/12/04 04:04:34 Starting bucket ops to source bucket
2021/12/04 04:04:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:04:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:04:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:04:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:04:39 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 04:04:44 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 04:04:49 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 04:04:54 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 04:04:54 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/04 04:04:54 No eventing-con process running
2021/12/04 04:04:54 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/04 04:04:56 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:05:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:05:03 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:05:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:05:08 Pumping document in the destination bucket
2021/12/04 04:05:08 Starting bucket ops to source bucket
2021/12/04 04:05:09 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:05:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/04 04:05:15 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 04:05:20 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 04:05:25 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 04:05:30 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 04:05:30 App: TestAdvancedInsertKeyExist got deployed
2021/12/04 04:05:30 Starting bucket ops to source bucket
2021/12/04 04:05:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:05:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:05:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:05:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:05:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:05:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:05:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:05:50 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 04:05:55 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 04:06:00 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 04:06:05 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 04:06:05 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/04 04:06:05 No eventing-con process running
2021/12/04 04:06:05 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/04 04:06:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:06:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:06:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:06:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:06:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.94s)
=== RUN   TestExipryGet
2021/12/04 04:06:24 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:06:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/04 04:06:30 Waiting for app: TestExipryGet to get deployed
2021/12/04 04:06:35 Waiting for app: TestExipryGet to get deployed
2021/12/04 04:06:40 Waiting for app: TestExipryGet to get deployed
2021/12/04 04:06:45 Waiting for app: TestExipryGet to get deployed
2021/12/04 04:06:45 App: TestExipryGet got deployed
2021/12/04 04:06:45 Starting bucket ops to source bucket
2021/12/04 04:07:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:07:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:07:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:07:30 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:07:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:07:35 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 04:07:40 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 04:07:45 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 04:07:50 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 04:07:50 App: TestExipryGet got un-deployed
2021/12/04 04:07:50 No eventing-con process running
2021/12/04 04:07:50 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/04 04:07:52 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:07:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:07:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:08:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:08:04 Pumping document in the destination bucket for expiry get
2021/12/04 04:08:04 Starting bucket ops to source bucket
2021/12/04 04:08:04 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:08:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/04 04:08:11 Waiting for app: TestExipryGet to get deployed
2021/12/04 04:08:16 Waiting for app: TestExipryGet to get deployed
2021/12/04 04:08:21 Waiting for app: TestExipryGet to get deployed
2021/12/04 04:08:26 Waiting for app: TestExipryGet to get deployed
2021/12/04 04:08:26 App: TestExipryGet got deployed
2021/12/04 04:08:26 Starting bucket ops to source bucket
2021/12/04 04:09:11 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:09:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:09:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:09:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:09:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:09:16 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 04:09:21 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 04:09:26 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 04:09:31 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 04:09:31 App: TestExipryGet got un-deployed
2021/12/04 04:09:31 No eventing-con process running
2021/12/04 04:09:31 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/04 04:09:33 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:09:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:09:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:09:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:09:50 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:09:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (211.06s)
=== RUN   TestExipryInsert
2021/12/04 04:09:55 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:09:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/04 04:10:01 Waiting for app: TestExipryInsert to get deployed
2021/12/04 04:10:06 Waiting for app: TestExipryInsert to get deployed
2021/12/04 04:10:11 Waiting for app: TestExipryInsert to get deployed
2021/12/04 04:10:16 Waiting for app: TestExipryInsert to get deployed
2021/12/04 04:10:16 App: TestExipryInsert got deployed
2021/12/04 04:10:16 Starting bucket ops to source bucket
2021/12/04 04:11:01 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/04 04:11:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:11:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:11:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:11:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:11:06 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 04:11:11 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 04:11:16 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 04:11:21 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 04:11:21 App: TestExipryInsert got un-deployed
2021/12/04 04:11:21 No eventing-con process running
2021/12/04 04:11:21 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/04 04:11:23 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:11:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:11:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:11:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:11:35 Testing on the destination bucket for expiry insert
2021/12/04 04:11:35 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:11:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/04 04:11:41 Waiting for app: TestExipryInsert to get deployed
2021/12/04 04:11:46 Waiting for app: TestExipryInsert to get deployed
2021/12/04 04:11:51 Waiting for app: TestExipryInsert to get deployed
2021/12/04 04:11:57 Waiting for app: TestExipryInsert to get deployed
2021/12/04 04:11:57 App: TestExipryInsert got deployed
2021/12/04 04:11:57 Starting bucket ops to source bucket
2021/12/04 04:12:42 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:12:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:12:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:12:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:12:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:12:47 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 04:12:52 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 04:12:57 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 04:13:02 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 04:13:02 App: TestExipryInsert got un-deployed
2021/12/04 04:13:02 No eventing-con process running
2021/12/04 04:13:02 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/04 04:13:04 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:13:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:13:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:13:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:13:21 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:13:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.95s)
=== RUN   TestExipryReplace
2021/12/04 04:13:26 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:13:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 04:13:32 Waiting for app: TestExipryReplace to get deployed
2021/12/04 04:13:37 Waiting for app: TestExipryReplace to get deployed
2021/12/04 04:13:42 Waiting for app: TestExipryReplace to get deployed
2021/12/04 04:13:47 Waiting for app: TestExipryReplace to get deployed
2021/12/04 04:13:47 App: TestExipryReplace got deployed
2021/12/04 04:13:47 Starting bucket ops to source bucket
2021/12/04 04:14:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:14:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:14:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:14:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:14:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:14:37 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 04:14:42 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 04:14:47 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 04:14:52 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 04:14:52 App: TestExipryReplace got un-deployed
2021/12/04 04:14:52 No eventing-con process running
2021/12/04 04:14:52 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/04 04:14:54 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:14:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:15:01 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:15:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:15:06 Testing on the destination bucket for expiry insert
2021/12/04 04:15:06 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:15:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 04:15:12 Waiting for app: TestExipryReplace to get deployed
2021/12/04 04:15:17 Waiting for app: TestExipryReplace to get deployed
2021/12/04 04:15:22 Waiting for app: TestExipryReplace to get deployed
2021/12/04 04:15:27 Waiting for app: TestExipryReplace to get deployed
2021/12/04 04:15:27 App: TestExipryReplace got deployed
2021/12/04 04:15:27 Starting bucket ops to source bucket
2021/12/04 04:16:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:16:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:16:23 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:16:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:16:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:16:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:16:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:16:28 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 04:16:33 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 04:16:38 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 04:16:43 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 04:16:43 App: TestExipryReplace got un-deployed
2021/12/04 04:16:43 No eventing-con process running
2021/12/04 04:16:43 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/04 04:16:45 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:16:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:16:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:16:56 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:17:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (215.88s)
=== RUN   TestExipryUpsert
2021/12/04 04:17:01 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:17:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/04 04:17:08 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 04:17:13 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 04:17:18 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 04:17:23 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 04:17:23 App: TestExipryUpsert got deployed
2021/12/04 04:17:23 Starting bucket ops to source bucket
2021/12/04 04:18:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:18:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:18:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:18:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:18:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:18:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:18:18 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 04:18:23 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 04:18:28 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 04:18:33 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 04:18:33 App: TestExipryUpsert got un-deployed
2021/12/04 04:18:33 No eventing-con process running
2021/12/04 04:18:33 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/04 04:18:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:18:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:18:42 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:18:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:18:47 Testing on the destination bucket for expiry insert
2021/12/04 04:18:47 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:18:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/04 04:18:53 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 04:18:58 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 04:19:03 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 04:19:08 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 04:19:08 App: TestExipryUpsert got deployed
2021/12/04 04:19:08 Starting bucket ops to source bucket
2021/12/04 04:19:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:19:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:19:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:19:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:19:59 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:19:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:20:04 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 04:20:09 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 04:20:14 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 04:20:19 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 04:20:19 App: TestExipryUpsert got un-deployed
2021/12/04 04:20:19 No eventing-con process running
2021/12/04 04:20:19 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/04 04:20:20 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:20:30 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/04 04:20:35 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/04 04:20:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:20:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:20:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.81s)
=== RUN   TestCasReplace
2021/12/04 04:20:42 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:20:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 04:20:48 Waiting for app: TestCasReplace to get deployed
2021/12/04 04:20:54 Waiting for app: TestCasReplace to get deployed
2021/12/04 04:20:59 Waiting for app: TestCasReplace to get deployed
2021/12/04 04:21:04 Waiting for app: TestCasReplace to get deployed
2021/12/04 04:21:04 App: TestCasReplace got deployed
2021/12/04 04:21:04 Starting bucket ops to source bucket
2021/12/04 04:21:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:21:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:21:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:21:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:21:09 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 04:21:14 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 04:21:19 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 04:21:24 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 04:21:24 App: TestCasReplace got un-deployed
2021/12/04 04:21:24 No eventing-con process running
2021/12/04 04:21:24 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/04 04:21:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:21:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:21:33 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:21:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:21:38 Testing on the destination bucket for upsert with cas
2021/12/04 04:21:38 Starting bucket ops to source bucket
2021/12/04 04:21:38 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:21:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 04:21:44 Waiting for app: TestCasReplace to get deployed
2021/12/04 04:21:49 Waiting for app: TestCasReplace to get deployed
2021/12/04 04:21:54 Waiting for app: TestCasReplace to get deployed
2021/12/04 04:21:59 Waiting for app: TestCasReplace to get deployed
2021/12/04 04:21:59 App: TestCasReplace got deployed
2021/12/04 04:21:59 Starting bucket ops to source bucket
2021/12/04 04:22:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:22:10 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:22:15 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:22:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:22:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:22:15 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:22:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:22:20 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 04:22:25 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 04:22:30 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 04:22:35 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 04:22:35 App: TestCasReplace got un-deployed
2021/12/04 04:22:35 No eventing-con process running
2021/12/04 04:22:35 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/04 04:22:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:22:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:22:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:22:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:22:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.99s)
=== RUN   TestCasDelete
2021/12/04 04:22:53 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:22:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 04:22:59 Waiting for app: TestCasDelete to get deployed
2021/12/04 04:23:04 Waiting for app: TestCasDelete to get deployed
2021/12/04 04:23:09 Waiting for app: TestCasDelete to get deployed
2021/12/04 04:23:14 Waiting for app: TestCasDelete to get deployed
2021/12/04 04:23:14 App: TestCasDelete got deployed
2021/12/04 04:23:14 Starting bucket ops to source bucket
2021/12/04 04:23:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:23:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:23:15 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:23:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:23:20 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 04:23:25 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 04:23:30 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 04:23:35 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 04:23:35 App: TestCasDelete got un-deployed
2021/12/04 04:23:35 No eventing-con process running
2021/12/04 04:23:35 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/04 04:23:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:23:47 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/04 04:23:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:23:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:23:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:23:53 Testing on the destination bucket for delete with cas
2021/12/04 04:23:53 Starting bucket ops to source bucket
2021/12/04 04:23:54 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:23:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 04:24:00 Waiting for app: TestCasDelete to get deployed
2021/12/04 04:24:05 Waiting for app: TestCasDelete to get deployed
2021/12/04 04:24:10 Waiting for app: TestCasDelete to get deployed
2021/12/04 04:24:15 Waiting for app: TestCasDelete to get deployed
2021/12/04 04:24:15 App: TestCasDelete got deployed
2021/12/04 04:24:15 Starting bucket ops to source bucket
2021/12/04 04:24:20 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:24:25 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:24:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:24:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:24:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 04:24:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:24:30 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 04:24:35 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 04:24:40 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 04:24:45 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 04:24:45 App: TestCasDelete got un-deployed
2021/12/04 04:24:45 No eventing-con process running
2021/12/04 04:24:45 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/04 04:24:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:24:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:24:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:24:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:25:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.44s)
=== RUN   TestCountersIncrement
2021/12/04 04:25:04 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 04:25:10 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 04:25:10 Removing node(s): [] from the cluster
2021/12/04 04:25:15 Rebalance progress: 100
2021/12/04 04:25:15 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:25:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/04 04:25:21 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 04:25:26 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 04:25:31 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 04:25:37 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 04:25:37 App: TestCountersIncrement got deployed
2021/12/04 04:25:37 Starting bucket ops to source bucket
2021/12/04 04:25:43 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/04 04:25:48 Waiting for dst bucket item count to get to: 1024 curr count: 311
2021/12/04 04:25:53 Waiting for dst bucket item count to get to: 1024 curr count: 823
2021/12/04 04:25:53 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/04 04:25:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:25:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:25:58 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 04:26:03 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 04:26:08 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 04:26:13 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 04:26:13 App: TestCountersIncrement got un-deployed
2021/12/04 04:26:13 No eventing-con process running
2021/12/04 04:26:13 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/04 04:26:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:26:25 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/04 04:26:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:26:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:26:36 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 04:26:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:26:36 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:26:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/04 04:26:43 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 04:26:48 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 04:26:53 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 04:26:58 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 04:26:58 App: TestCountersIncrement got deployed
2021/12/04 04:26:58 Starting bucket ops to source bucket
2021/12/04 04:27:04 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/04 04:27:09 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/04 04:27:14 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/04 04:27:14 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/04 04:27:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:27:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:27:19 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 04:27:24 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 04:27:29 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 04:27:34 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 04:27:34 App: TestCountersIncrement got un-deployed
2021/12/04 04:27:34 No eventing-con process running
2021/12/04 04:27:34 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/04 04:27:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:27:46 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/04 04:27:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:27:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:27:58 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/04 04:27:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:27:58 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/04 04:28:03 Rebalance progress: 100
--- PASS: TestCountersIncrement (179.01s)
=== RUN   TestCountersDecrement
2021/12/04 04:28:03 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 04:28:03 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 04:28:03 Removing node(s): [] from the cluster
2021/12/04 04:28:08 Rebalance progress: 100
2021/12/04 04:28:08 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:28:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/04 04:28:15 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 04:28:20 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 04:28:25 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 04:28:30 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 04:28:30 App: TestCountersDecrement got deployed
2021/12/04 04:28:30 Starting bucket ops to source bucket
2021/12/04 04:28:36 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/04 04:28:41 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/04 04:28:46 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/04 04:28:46 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/04 04:28:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:28:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:28:51 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 04:28:56 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 04:29:01 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 04:29:06 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 04:29:06 App: TestCountersDecrement got un-deployed
2021/12/04 04:29:06 No eventing-con process running
2021/12/04 04:29:06 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/04 04:29:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:29:18 Waiting for dst bucket item count to get to: 0 curr count: 529
2021/12/04 04:29:23 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/04 04:29:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:29:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:29:34 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 04:29:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:29:35 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:29:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/04 04:29:41 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 04:29:46 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 04:29:51 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 04:29:56 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 04:29:56 App: TestCountersDecrement got deployed
2021/12/04 04:29:56 Starting bucket ops to source bucket
2021/12/04 04:30:02 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/04 04:30:02 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/04 04:30:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:30:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:30:07 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 04:30:12 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 04:30:17 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 04:30:22 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 04:30:22 App: TestCountersDecrement got un-deployed
2021/12/04 04:30:22 No eventing-con process running
2021/12/04 04:30:22 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/04 04:30:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:30:34 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/04 04:30:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 04:30:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:30:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:30:51 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/04 04:30:56 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/04 04:30:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:30:56 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/04 04:31:01 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.19s)
=== RUN   TestCacheGet
2021/12/04 04:31:01 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:31:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/04 04:31:07 Waiting for app: TestCacheGet to get deployed
2021/12/04 04:31:12 Waiting for app: TestCacheGet to get deployed
2021/12/04 04:31:17 Waiting for app: TestCacheGet to get deployed
2021/12/04 04:31:22 Waiting for app: TestCacheGet to get deployed
2021/12/04 04:31:22 App: TestCacheGet got deployed
2021/12/04 04:31:22 Starting bucket ops to source bucket
2021/12/04 04:31:27 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/04 04:31:32 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/04 04:31:37 Waiting for dst bucket item count to get to: 20 curr count: 3
2021/12/04 04:31:42 Waiting for dst bucket item count to get to: 20 curr count: 3
2021/12/04 04:31:47 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/04 04:31:52 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/04 04:31:52 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/04 04:31:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:31:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:31:57 Waiting for app: TestCacheGet to get un-deployed
2021/12/04 04:32:03 Waiting for app: TestCacheGet to get un-deployed
2021/12/04 04:32:08 Waiting for app: TestCacheGet to get un-deployed
2021/12/04 04:32:13 Waiting for app: TestCacheGet to get un-deployed
2021/12/04 04:32:13 App: TestCacheGet got un-deployed
2021/12/04 04:32:13 No eventing-con process running
2021/12/04 04:32:13 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/04 04:32:14 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:32:24 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/04 04:32:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:32:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:32:36 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/04 04:32:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.27s)
=== RUN   TestCacheRYOW
2021/12/04 04:32:36 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:32:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/04 04:32:42 Waiting for app: TestCacheRYOW to get deployed
2021/12/04 04:32:47 Waiting for app: TestCacheRYOW to get deployed
2021/12/04 04:32:52 Waiting for app: TestCacheRYOW to get deployed
2021/12/04 04:32:57 Waiting for app: TestCacheRYOW to get deployed
2021/12/04 04:32:57 App: TestCacheRYOW got deployed
2021/12/04 04:32:57 Starting bucket ops to source bucket
2021/12/04 04:33:03 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/04 04:33:08 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/04 04:33:13 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/04 04:33:13 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/04 04:33:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:33:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:33:18 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/04 04:33:23 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/04 04:33:28 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/04 04:33:33 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/04 04:33:33 App: TestCacheRYOW got un-deployed
2021/12/04 04:33:33 No eventing-con process running
2021/12/04 04:33:33 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/04 04:33:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:33:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:33:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:33:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:33:56 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:33:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.28s)
=== RUN   TestCacheExpiry
2021/12/04 04:33:56 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:33:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/04 04:34:03 Waiting for app: TestCacheExpiry to get deployed
2021/12/04 04:34:08 Waiting for app: TestCacheExpiry to get deployed
2021/12/04 04:34:13 Waiting for app: TestCacheExpiry to get deployed
2021/12/04 04:34:18 Waiting for app: TestCacheExpiry to get deployed
2021/12/04 04:34:18 App: TestCacheExpiry got deployed
2021/12/04 04:34:18 Starting bucket ops to source bucket
2021/12/04 04:34:23 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 04:34:28 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/04 04:34:33 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/04 04:34:38 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/04 04:34:43 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/04 04:34:43 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 04:34:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:34:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:34:48 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/04 04:34:53 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/04 04:34:58 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/04 04:35:03 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/04 04:35:03 App: TestCacheExpiry got un-deployed
2021/12/04 04:35:03 No eventing-con process running
2021/12/04 04:35:03 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/04 04:35:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:35:15 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:35:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:35:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:35:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:35:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (90.27s)
=== RUN   TestCacheExpiryCustom
2021/12/04 04:35:27 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:35:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/04 04:35:33 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/04 04:35:38 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/04 04:35:43 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/04 04:35:48 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/04 04:35:48 App: TestCacheExpiryCustom got deployed
2021/12/04 04:35:48 Starting bucket ops to source bucket
2021/12/04 04:35:53 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 04:35:58 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/04 04:36:03 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/04 04:36:08 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/04 04:36:13 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/04 04:36:13 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 04:36:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:36:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:36:18 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/04 04:36:23 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/04 04:36:28 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/04 04:36:33 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/04 04:36:33 App: TestCacheExpiryCustom got un-deployed
2021/12/04 04:36:33 No eventing-con process running
2021/12/04 04:36:33 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/04 04:36:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:36:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:36:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:36:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:36:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:36:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (90.20s)
=== RUN   TestCacheOverflowCustom
2021/12/04 04:36:57 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:36:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/04 04:37:03 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/04 04:37:08 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/04 04:37:13 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/04 04:37:18 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/04 04:37:18 App: TestCacheOverflowCustom got deployed
2021/12/04 04:37:18 Starting bucket ops to source bucket
2021/12/04 04:37:23 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/04 04:37:28 Waiting for dst bucket item count to get to: 35 curr count: 2
2021/12/04 04:37:33 Waiting for dst bucket item count to get to: 35 curr count: 13
2021/12/04 04:37:38 Waiting for dst bucket item count to get to: 35 curr count: 24
2021/12/04 04:37:43 Waiting for dst bucket item count to get to: 35 curr count: 30
2021/12/04 04:37:48 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/04 04:37:53 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/04 04:37:53 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/04 04:37:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:37:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:37:58 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/04 04:38:03 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/04 04:38:08 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/04 04:38:13 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/04 04:38:13 App: TestCacheOverflowCustom got un-deployed
2021/12/04 04:38:13 No eventing-con process running
2021/12/04 04:38:14 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/04 04:38:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:38:25 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/04 04:38:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:38:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:38:37 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/04 04:38:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.24s)
=== RUN   TestStrictMode
2021/12/04 04:38:37 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:38:38 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/04 04:38:38 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:38:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/04 04:38:44 Waiting for app: TestDataTypes to get deployed
2021/12/04 04:38:49 Waiting for app: TestDataTypes to get deployed
2021/12/04 04:38:54 Waiting for app: TestDataTypes to get deployed
2021/12/04 04:38:59 Waiting for app: TestDataTypes to get deployed
2021/12/04 04:38:59 App: TestDataTypes got deployed
2021/12/04 04:38:59 Starting bucket ops to source bucket
2021/12/04 04:39:04 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/04 04:39:09 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/04 04:39:15 Waiting for dst bucket item count to get to: 18 curr count: 8
2021/12/04 04:39:15 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/04 04:39:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:39:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:39:20 Waiting for app: TestDataTypes to get un-deployed
2021/12/04 04:39:25 Waiting for app: TestDataTypes to get un-deployed
2021/12/04 04:39:30 Waiting for app: TestDataTypes to get un-deployed
2021/12/04 04:39:35 Waiting for app: TestDataTypes to get un-deployed
2021/12/04 04:39:35 App: TestDataTypes got un-deployed
2021/12/04 04:39:35 No eventing-con process running
2021/12/04 04:39:35 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/04 04:39:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:39:47 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/04 04:39:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:39:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:39:58 Waiting for dst bucket item count to get to: 0 curr count: 15
2021/12/04 04:40:03 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/04 04:40:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (85.21s)
=== RUN   TestEnoentGet
2021/12/04 04:40:03 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:40:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/04 04:40:10 Waiting for app: TestEnoentGet to get deployed
2021/12/04 04:40:15 Waiting for app: TestEnoentGet to get deployed
2021/12/04 04:40:20 Waiting for app: TestEnoentGet to get deployed
2021/12/04 04:40:25 Waiting for app: TestEnoentGet to get deployed
2021/12/04 04:40:25 App: TestEnoentGet got deployed
2021/12/04 04:40:25 Starting bucket ops to source bucket
2021/12/04 04:40:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 04:40:35 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/04 04:40:35 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 04:40:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:40:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:40:40 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 04:40:45 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 04:40:50 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 04:40:55 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 04:40:55 App: TestEnoentGet got un-deployed
2021/12/04 04:40:55 No eventing-con process running
2021/12/04 04:40:55 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/04 04:40:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:41:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:41:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:41:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:41:19 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/04 04:41:24 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/04 04:41:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:41:24 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:41:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/04 04:41:30 Waiting for app: TestEnoentGet to get deployed
2021/12/04 04:41:35 Waiting for app: TestEnoentGet to get deployed
2021/12/04 04:41:40 Waiting for app: TestEnoentGet to get deployed
2021/12/04 04:41:45 Waiting for app: TestEnoentGet to get deployed
2021/12/04 04:41:45 App: TestEnoentGet got deployed
2021/12/04 04:41:45 Starting bucket ops to source bucket
2021/12/04 04:41:50 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 04:41:55 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/04 04:41:55 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 04:41:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:41:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:42:00 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 04:42:05 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 04:42:10 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 04:42:15 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 04:42:15 App: TestEnoentGet got un-deployed
2021/12/04 04:42:15 No eventing-con process running
2021/12/04 04:42:15 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/04 04:42:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:42:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:42:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:42:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:42:39 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/04 04:42:44 Waiting for dst bucket item count to get to: 0 curr count: 41
2021/12/04 04:42:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.36s)
=== RUN   TestEnoentDelete
2021/12/04 04:42:44 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:42:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 04:42:50 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 04:42:55 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 04:43:00 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 04:43:05 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 04:43:05 App: TestEnoentDelete got deployed
2021/12/04 04:43:05 Starting bucket ops to source bucket
2021/12/04 04:43:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 04:43:15 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/04 04:43:15 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 04:43:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:43:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:43:20 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 04:43:25 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 04:43:30 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 04:43:35 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 04:43:35 App: TestEnoentDelete got un-deployed
2021/12/04 04:43:35 No eventing-con process running
2021/12/04 04:43:35 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/04 04:43:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:43:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:43:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:43:49 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:43:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:44:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:44:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:44:04 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:44:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 04:44:10 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 04:44:15 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 04:44:20 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 04:44:25 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 04:44:25 App: TestEnoentDelete got deployed
2021/12/04 04:44:25 Starting bucket ops to source bucket
2021/12/04 04:44:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 04:44:30 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 04:44:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:44:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:44:36 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 04:44:41 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 04:44:46 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 04:44:51 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 04:44:51 App: TestEnoentDelete got un-deployed
2021/12/04 04:44:51 No eventing-con process running
2021/12/04 04:44:51 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/04 04:44:52 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:45:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:45:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:45:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:45:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:45:19 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:45:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 04:45:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.44s)
=== RUN   TestError
2021/12/04 04:45:34 Waiting for app: TestError to get un-deployed
2021/12/04 04:45:34 App: TestError got un-deployed
2021/12/04 04:45:34 No eventing-con process running
2021/12/04 04:45:34 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/04 04:45:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:45:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:45:43 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:45:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:45:48 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:45:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/04 04:45:54 Waiting for app: TestError to get deployed
2021/12/04 04:45:59 Waiting for app: TestError to get deployed
2021/12/04 04:46:04 Waiting for app: TestError to get deployed
2021/12/04 04:46:09 Waiting for app: TestError to get deployed
2021/12/04 04:46:09 App: TestError got deployed
2021/12/04 04:46:09 Starting bucket ops to source bucket
2021/12/04 04:46:14 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/04 04:46:19 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/04 04:46:19 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/04 04:46:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:46:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:46:24 Waiting for app: TestError to get un-deployed
2021/12/04 04:46:29 Waiting for app: TestError to get un-deployed
2021/12/04 04:46:34 Waiting for app: TestError to get un-deployed
2021/12/04 04:46:39 Waiting for app: TestError to get un-deployed
2021/12/04 04:46:39 App: TestError got un-deployed
2021/12/04 04:46:40 No eventing-con process running
2021/12/04 04:46:40 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/04 04:46:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:46:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:46:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:46:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:46:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:47:08 Waiting for dst bucket item count to get to: 0 curr count: 229
2021/12/04 04:47:13 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/04 04:47:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.96s)
=== RUN   TestCRLF
2021/12/04 04:47:23 Waiting for app: TestCRLF to get un-deployed
2021/12/04 04:47:23 App: TestCRLF got un-deployed
2021/12/04 04:47:23 No eventing-con process running
2021/12/04 04:47:23 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/04 04:47:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:47:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:47:32 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:47:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:47:37 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:47:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/04 04:47:43 Waiting for app: TestCRLF to get deployed
2021/12/04 04:47:48 Waiting for app: TestCRLF to get deployed
2021/12/04 04:47:53 Waiting for app: TestCRLF to get deployed
2021/12/04 04:47:58 Waiting for app: TestCRLF to get deployed
2021/12/04 04:47:58 App: TestCRLF got deployed
2021/12/04 04:47:58 Starting bucket ops to source bucket
2021/12/04 04:48:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 04:48:08 Waiting for dst bucket item count to get to: 100 curr count: 49
2021/12/04 04:48:08 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 04:48:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:48:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:48:13 Waiting for app: TestCRLF to get un-deployed
2021/12/04 04:48:19 Waiting for app: TestCRLF to get un-deployed
2021/12/04 04:48:24 Waiting for app: TestCRLF to get un-deployed
2021/12/04 04:48:29 Waiting for app: TestCRLF to get un-deployed
2021/12/04 04:48:29 App: TestCRLF got un-deployed
2021/12/04 04:48:29 No eventing-con process running
2021/12/04 04:48:29 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/04 04:48:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:48:40 Waiting for dst bucket item count to get to: 0 curr count: 77
2021/12/04 04:48:45 Waiting for dst bucket item count to get to: 0 curr count: 77
2021/12/04 04:48:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:48:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:48:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 04:48:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (104.08s)
=== RUN   TestImportExport
2021/12/04 04:49:07 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 04:49:07 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/04 04:49:07 No eventing-con process running
2021/12/04 04:49:07 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/04 04:49:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:49:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:49:16 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:49:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:49:26 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 04:49:26 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/04 04:49:26 No eventing-con process running
2021/12/04 04:49:26 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/04 04:49:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:49:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:49:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:49:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:49:40 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:49:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 04:49:41 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:49:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 04:49:47 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/04 04:49:52 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/04 04:49:57 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/04 04:50:02 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/04 04:50:02 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/04 04:50:07 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/04 04:50:12 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/04 04:50:17 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/04 04:50:17 App: TestImportExport_bucket_op_on_update got deployed
2021/12/04 04:50:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:50:22 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 04:50:27 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 04:50:32 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 04:50:37 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 04:50:37 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/04 04:50:37 No eventing-con process running
2021/12/04 04:50:37 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/04 04:50:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:50:43 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 04:50:48 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 04:50:53 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 04:50:58 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 04:50:58 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/04 04:50:58 No eventing-con process running
2021/12/04 04:50:58 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/04 04:51:15 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 04:51:15 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/04 04:51:15 No eventing-con process running
2021/12/04 04:51:15 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/04 04:51:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:51:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:51:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:51:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:51:34 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 04:51:34 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/04 04:51:34 No eventing-con process running
2021/12/04 04:51:34 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/04 04:51:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:51:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:51:42 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:51:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (170.00s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/04 04:51:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:51:57 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 04:51:57 No eventing-con process running
2021/12/04 04:51:57 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/04 04:51:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:52:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:52:06 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:52:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:52:11 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:52:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 04:52:12 Starting bucket ops to source bucket
2021/12/04 04:52:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:52:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:52:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:52:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:52:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 04:52:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/04 04:52:40 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/04 04:52:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:52:40 Undeploying app: bucket_op_on_update
2021/12/04 04:52:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:52:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:52:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:52:55 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:53:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:53:00 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 04:53:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:53:00 No eventing-con process running
2021/12/04 04:53:02 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:53:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:53:45 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:53:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 04:53:46 Starting bucket ops to source bucket
2021/12/04 04:53:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:53:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:54:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:54:09 Waiting for dst bucket item count to get to: 5000 curr count: 2505
2021/12/04 04:54:14 Waiting for dst bucket item count to get to: 5000 curr count: 2505
2021/12/04 04:54:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 04:54:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/04 04:54:19 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/04 04:54:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:54:19 Undeploying app: bucket_op_on_update
2021/12/04 04:54:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:54:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:54:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:54:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:54:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:54:39 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 04:54:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:54:39 No eventing-con process running
2021/12/04 04:54:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:54:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:55:23 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:55:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 04:55:24 Starting bucket ops to source bucket
2021/12/04 04:55:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:55:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:55:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:55:47 Waiting for dst bucket item count to get to: 5000 curr count: 1627
2021/12/04 04:55:52 Waiting for dst bucket item count to get to: 5000 curr count: 4127
2021/12/04 04:55:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 04:55:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/04 04:55:57 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/04 04:55:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:55:57 Undeploying app: bucket_op_on_update
2021/12/04 04:55:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:56:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:56:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:56:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:56:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:56:17 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 04:56:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:56:17 No eventing-con process running
2021/12/04 04:56:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:56:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:57:02 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:57:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 04:57:03 Starting bucket ops to source bucket
2021/12/04 04:57:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:57:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:57:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:57:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:57:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 04:57:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/04 04:57:31 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/04 04:57:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:57:31 Undeploying app: bucket_op_on_update
2021/12/04 04:57:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:57:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:57:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:57:46 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:57:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:57:51 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 04:57:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:57:51 No eventing-con process running
2021/12/04 04:57:53 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:58:00 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:58:35 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 04:58:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 04:58:36 Starting bucket ops to source bucket
2021/12/04 04:58:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:58:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:58:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 04:58:59 Waiting for dst bucket item count to get to: 5000 curr count: 1393
2021/12/04 04:59:04 Waiting for dst bucket item count to get to: 5000 curr count: 1393
2021/12/04 04:59:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 04:59:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/04 04:59:09 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/04 04:59:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 04:59:09 Undeploying app: bucket_op_on_update
2021/12/04 04:59:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 04:59:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:59:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:59:25 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:59:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 04:59:30 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 04:59:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 04:59:30 No eventing-con process running
2021/12/04 04:59:31 Context: Bucket flush, response code: 200 dump: 
2021/12/04 04:59:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:00:14 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (506.59s)
=== RUN   TestOnUpdateN1QLOp
2021/12/04 05:00:24 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/04 05:00:24 App: TestOnUpdateN1QLOp got un-deployed
2021/12/04 05:00:24 No eventing-con process running
2021/12/04 05:00:24 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/04 05:00:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:00:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:00:32 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:00:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:00:37 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:00:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/04 05:00:39 Starting bucket ops to source bucket
2021/12/04 05:00:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:00:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:00:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:01:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:01:07 Waiting for dst bucket item count to get to: 5000 curr count: 2463
2021/12/04 05:01:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:01:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:01:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:01:12 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/04 05:01:17 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/04 05:01:22 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/04 05:01:27 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/04 05:01:27 App: TestOnUpdateN1QLOp got un-deployed
2021/12/04 05:01:27 No eventing-con process running
2021/12/04 05:01:27 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/04 05:01:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:01:39 Waiting for dst bucket item count to get to: 0 curr count: 4662
2021/12/04 05:01:44 Waiting for dst bucket item count to get to: 0 curr count: 2162
2021/12/04 05:01:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:01:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:01:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:01:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.78s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/04 05:02:06 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/04 05:02:06 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/04 05:02:06 No eventing-con process running
2021/12/04 05:02:06 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/04 05:02:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:02:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:02:14 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:02:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:02:19 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:02:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 05:02:20 Starting bucket ops to source bucket
2021/12/04 05:02:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:02:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:02:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:02:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:02:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:02:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:02:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:02:48 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/04 05:02:54 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/04 05:02:59 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/04 05:03:04 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/04 05:03:04 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/04 05:03:04 No eventing-con process running
2021/12/04 05:03:04 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/04 05:03:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:03:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:03:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:03:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:03:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:03:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (91.64s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/04 05:03:37 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 05:03:37 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/04 05:03:37 No eventing-con process running
2021/12/04 05:03:37 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/04 05:03:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:03:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:03:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:03:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:03:51 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:03:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 05:03:52 Starting bucket ops to source bucket
2021/12/04 05:04:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:04:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:04:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:04:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:04:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:04:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:04:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:04:20 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 05:04:25 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 05:04:30 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 05:04:35 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 05:04:35 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/04 05:04:35 No eventing-con process running
2021/12/04 05:04:35 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/04 05:04:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:04:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:04:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:04:49 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:04:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:05:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 05:05:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (96.74s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/04 05:05:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/04 05:05:14 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/04 05:05:14 No eventing-con process running
2021/12/04 05:05:14 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/04 05:05:16 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:05:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:05:23 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:05:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:05:28 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:05:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/04 05:05:29 Starting bucket ops to source bucket
2021/12/04 05:05:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:05:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:05:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:05:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:05:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/04 05:05:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:05:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:05:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:06:02 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/04 05:06:07 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/04 05:06:12 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/04 05:06:17 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/04 05:06:17 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/04 05:06:17 No eventing-con process running
2021/12/04 05:06:17 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/04 05:06:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:06:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:06:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 05:06:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:06:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:06:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:06:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.74s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/04 05:06:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/04 05:06:56 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/04 05:06:56 No eventing-con process running
2021/12/04 05:06:56 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/04 05:06:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:07:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:07:04 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:07:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:07:10 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:07:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/04 05:07:11 Starting bucket ops to source bucket
2021/12/04 05:07:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:07:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:07:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:07:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:07:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:07:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:07:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:07:39 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/04 05:07:44 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/04 05:07:49 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/04 05:07:54 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/04 05:07:54 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/04 05:07:54 No eventing-con process running
2021/12/04 05:07:54 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/04 05:07:56 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:08:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:08:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:08:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:08:18 Waiting for dst bucket item count to get to: 0 curr count: 4136
2021/12/04 05:08:23 Waiting for dst bucket item count to get to: 0 curr count: 1636
2021/12/04 05:08:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.50s)
=== RUN   TestOnDeleteBucketOp
2021/12/04 05:08:33 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/04 05:08:33 App: TestOnDeleteBucketOp got un-deployed
2021/12/04 05:08:33 No eventing-con process running
2021/12/04 05:08:33 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/04 05:08:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:08:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:08:42 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:08:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:08:47 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:08:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 05:08:48 Starting bucket ops to source bucket
2021/12/04 05:08:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:09:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:09:09 Waiting for dst bucket item count to get to: 5000 curr count: 77
2021/12/04 05:09:14 Waiting for dst bucket item count to get to: 5000 curr count: 2581
2021/12/04 05:09:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:09:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:09:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:09:19 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/04 05:09:24 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/04 05:09:29 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/04 05:09:34 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/04 05:09:34 App: TestOnDeleteBucketOp got un-deployed
2021/12/04 05:09:34 No eventing-con process running
2021/12/04 05:09:34 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/04 05:09:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:09:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:09:43 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:09:53 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:09:58 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/04 05:09:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (95.18s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/04 05:10:08 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/04 05:10:08 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/04 05:10:08 No eventing-con process running
2021/12/04 05:10:08 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/04 05:10:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:10:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:10:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:10:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:10:22 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:10:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/04 05:10:23 Starting bucket ops to source bucket
2021/12/04 05:10:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:10:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:10:44 Waiting for dst bucket item count to get to: 5000 curr count: 2300
2021/12/04 05:10:49 Waiting for dst bucket item count to get to: 5000 curr count: 4796
2021/12/04 05:10:49 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:10:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:10:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:10:54 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/04 05:10:59 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/04 05:11:04 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/04 05:11:09 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/04 05:11:09 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/04 05:11:09 No eventing-con process running
2021/12/04 05:11:09 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/04 05:11:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:11:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:11:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:11:28 Waiting for dst bucket item count to get to: 0 curr count: 3681
2021/12/04 05:11:33 Waiting for dst bucket item count to get to: 0 curr count: 1185
2021/12/04 05:11:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (95.05s)
=== RUN   TestTimerBucketOp
2021/12/04 05:11:43 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/04 05:11:44 App: TestTimerBucketOp got un-deployed
2021/12/04 05:11:44 No eventing-con process running
2021/12/04 05:11:44 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/04 05:11:45 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:11:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:11:52 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:11:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:11:57 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:11:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 05:11:58 Starting bucket ops to source bucket
2021/12/04 05:12:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:12:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:12:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:12:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:12:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:12:31 Waiting for dst bucket item count to get to: 5000 curr count: 1143
2021/12/04 05:12:36 Waiting for dst bucket item count to get to: 5000 curr count: 3643
2021/12/04 05:12:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:12:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:12:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:12:41 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/04 05:12:46 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/04 05:12:51 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/04 05:12:56 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/04 05:12:56 App: TestTimerBucketOp got un-deployed
2021/12/04 05:12:56 No eventing-con process running
2021/12/04 05:12:57 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/04 05:12:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:13:08 Waiting for dst bucket item count to get to: 0 curr count: 3902
2021/12/04 05:13:13 Waiting for dst bucket item count to get to: 0 curr count: 1402
2021/12/04 05:13:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:13:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:13:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:13:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (111.72s)
=== RUN   TestTimerInPastBucketOp
2021/12/04 05:13:35 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/04 05:13:35 App: TestTimerInPastBucketOp got un-deployed
2021/12/04 05:13:35 No eventing-con process running
2021/12/04 05:13:35 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/04 05:13:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:13:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:13:44 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:13:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:13:49 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:13:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 05:13:50 Starting bucket ops to source bucket
2021/12/04 05:13:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:14:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:14:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:14:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:14:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:14:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:14:28 Waiting for dst bucket item count to get to: 5000 curr count: 973
2021/12/04 05:14:33 Waiting for dst bucket item count to get to: 5000 curr count: 3237
2021/12/04 05:14:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:14:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:14:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:14:38 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/04 05:14:43 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/04 05:14:48 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/04 05:14:53 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/04 05:14:53 App: TestTimerInPastBucketOp got un-deployed
2021/12/04 05:14:53 No eventing-con process running
2021/12/04 05:14:53 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/04 05:14:55 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:15:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:15:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:15:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:15:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:15:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.58s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/04 05:15:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:15:27 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 05:15:27 No eventing-con process running
2021/12/04 05:15:27 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/04 05:15:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:15:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:15:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:15:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:15:40 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:15:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 05:15:42 Starting bucket ops to source bucket
2021/12/04 05:15:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:15:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:16:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:16:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:16:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:16:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:16:20 Waiting for dst bucket item count to get to: 5000 curr count: 1440
2021/12/04 05:16:25 Waiting for dst bucket item count to get to: 5000 curr count: 1440
2021/12/04 05:16:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:16:30 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/04 05:16:30 App: TestDeployUndeployLoopTimer got deployed
2021/12/04 05:16:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:16:30 Undeploying app: bucket_op_with_timer
2021/12/04 05:16:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:16:35 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:16:40 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:16:45 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:16:50 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:16:50 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 05:16:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:16:50 No eventing-con process running
2021/12/04 05:16:51 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:16:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:17:34 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:17:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 05:17:35 Starting bucket ops to source bucket
2021/12/04 05:17:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:17:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:17:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:17:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:18:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:18:08 Waiting for dst bucket item count to get to: 5000 curr count: 1623
2021/12/04 05:18:13 Waiting for dst bucket item count to get to: 5000 curr count: 1623
2021/12/04 05:18:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:18:18 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/04 05:18:18 App: TestDeployUndeployLoopTimer got deployed
2021/12/04 05:18:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:18:18 Undeploying app: bucket_op_with_timer
2021/12/04 05:18:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:18:23 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:18:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:18:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:18:38 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:18:38 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 05:18:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:18:38 No eventing-con process running
2021/12/04 05:18:40 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:18:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:19:22 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:19:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 05:19:23 Starting bucket ops to source bucket
2021/12/04 05:19:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:19:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:19:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:19:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:19:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:19:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:20:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:20:06 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/04 05:20:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:20:11 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/04 05:20:11 App: TestDeployUndeployLoopTimer got deployed
2021/12/04 05:20:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:20:11 Undeploying app: bucket_op_with_timer
2021/12/04 05:20:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:20:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:20:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:20:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:20:32 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:20:32 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 05:20:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:20:32 No eventing-con process running
2021/12/04 05:20:33 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:20:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:21:16 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:21:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 05:21:17 Starting bucket ops to source bucket
2021/12/04 05:21:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:21:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:21:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:21:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:21:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:21:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:21:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:21:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:22:00 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/04 05:22:00 App: TestDeployUndeployLoopTimer got deployed
2021/12/04 05:22:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:22:00 Undeploying app: bucket_op_with_timer
2021/12/04 05:22:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:22:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:22:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:22:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:22:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:22:20 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 05:22:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:22:20 No eventing-con process running
2021/12/04 05:22:22 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:22:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:23:04 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:23:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 05:23:05 Starting bucket ops to source bucket
2021/12/04 05:23:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:23:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:23:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:23:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:23:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:23:38 Waiting for dst bucket item count to get to: 5000 curr count: 61
2021/12/04 05:23:43 Waiting for dst bucket item count to get to: 5000 curr count: 61
2021/12/04 05:23:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:23:48 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/04 05:23:48 App: TestDeployUndeployLoopTimer got deployed
2021/12/04 05:23:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:23:49 Undeploying app: bucket_op_with_timer
2021/12/04 05:23:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:23:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:23:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:24:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:24:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 05:24:09 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 05:24:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:24:09 No eventing-con process running
2021/12/04 05:24:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:24:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:24:53 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (576.08s)
=== RUN   TestMultipleHandlers
2021/12/04 05:25:03 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/04 05:25:03 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/04 05:25:03 No eventing-con process running
2021/12/04 05:25:03 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/04 05:25:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:25:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:25:12 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:25:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:25:22 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/04 05:25:22 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/04 05:25:22 No eventing-con process running
2021/12/04 05:25:22 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/04 05:25:23 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:25:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:25:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:25:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:25:35 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:25:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 05:25:36 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:25:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/04 05:25:42 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/04 05:25:47 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/04 05:25:53 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/04 05:25:58 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/04 05:25:58 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/04 05:26:03 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/04 05:26:08 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/04 05:26:13 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/04 05:26:13 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/04 05:26:13 Starting bucket ops to source bucket
2021/12/04 05:26:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:26:28 Waiting for dst bucket item count to get to: 5000 curr count: 4752
2021/12/04 05:26:33 Waiting for dst bucket item count to get to: 5000 curr count: 4752
2021/12/04 05:26:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:26:33 Starting bucket ops to source bucket
2021/12/04 05:26:48 Waiting for dst bucket item count to get to: 10000 curr count: 5752
2021/12/04 05:26:53 Waiting for dst bucket item count to get to: 10000 curr count: 5752
2021/12/04 05:26:53 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 05:26:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:26:53 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:26:53 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:26:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:26:58 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/04 05:27:03 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/04 05:27:08 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/04 05:27:13 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/04 05:27:13 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/04 05:27:13 No eventing-con process running
2021/12/04 05:27:13 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/04 05:27:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:27:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:27:23 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:27:33 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 05:27:38 Waiting for dst bucket item count to get to: 0 curr count: 4996
2021/12/04 05:27:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:27:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:27:43 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/04 05:27:48 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/04 05:27:53 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/04 05:27:58 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/04 05:27:58 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/04 05:27:58 No eventing-con process running
2021/12/04 05:27:58 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/04 05:28:00 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:28:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:28:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:28:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (198.66s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/04 05:28:22 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/04 05:28:22 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/04 05:28:22 No eventing-con process running
2021/12/04 05:28:22 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/04 05:28:23 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:28:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:28:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:28:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:28:35 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:28:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 05:28:36 Starting bucket ops to source bucket
2021/12/04 05:28:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:28:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:28:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:28:59 Waiting for dst bucket item count to get to: 5000 curr count: 1825
2021/12/04 05:29:04 Waiting for dst bucket item count to get to: 5000 curr count: 1825
2021/12/04 05:29:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:29:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:29:09 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/04 05:29:09 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/04 05:29:09 Pausing app: bucket_op_on_update
2021/12/04 05:29:09 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:29:14 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/04 05:29:14 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/04 05:29:14 Resuming app: bucket_op_on_update
2021/12/04 05:29:16 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:29:16 Starting bucket ops to source bucket
2021/12/04 05:29:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:29:29 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:29:34 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:29:34 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 05:29:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:29:39 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/04 05:29:39 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/04 05:29:39 Pausing app: bucket_op_on_update
2021/12/04 05:29:39 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:29:44 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/04 05:29:44 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/04 05:29:44 Resuming app: bucket_op_on_update
2021/12/04 05:29:45 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:29:45 Starting bucket ops to source bucket
2021/12/04 05:29:53 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 05:29:58 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 05:30:03 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 05:30:03 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/04 05:30:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:30:08 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/04 05:30:08 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/04 05:30:08 Pausing app: bucket_op_on_update
2021/12/04 05:30:08 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:30:13 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/04 05:30:13 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/04 05:30:13 Resuming app: bucket_op_on_update
2021/12/04 05:30:14 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:30:14 Starting bucket ops to source bucket
2021/12/04 05:30:22 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 05:30:27 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 05:30:32 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 05:30:32 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/04 05:30:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:30:37 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/04 05:30:37 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/04 05:30:37 Pausing app: bucket_op_on_update
2021/12/04 05:30:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:30:42 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/04 05:30:43 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/04 05:30:43 Resuming app: bucket_op_on_update
2021/12/04 05:30:44 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:30:44 Starting bucket ops to source bucket
2021/12/04 05:30:52 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 05:30:57 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 05:31:02 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 05:31:07 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/04 05:31:07 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/04 05:31:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:31:12 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/04 05:31:12 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/04 05:31:12 Pausing app: bucket_op_on_update
2021/12/04 05:31:12 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:31:17 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/04 05:31:17 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/04 05:31:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:31:22 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/04 05:31:27 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/04 05:31:32 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/04 05:31:37 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/04 05:31:37 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/04 05:31:37 No eventing-con process running
2021/12/04 05:31:37 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/04 05:31:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:31:49 Waiting for dst bucket item count to get to: 0 curr count: 24159
2021/12/04 05:31:54 Waiting for dst bucket item count to get to: 0 curr count: 11659
2021/12/04 05:31:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:31:56 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:32:06 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/04 05:32:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (234.24s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/04 05:32:16 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/04 05:32:16 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/04 05:32:16 No eventing-con process running
2021/12/04 05:32:16 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/04 05:32:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:32:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:32:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:32:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:32:29 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:32:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 05:32:31 Starting bucket ops to source bucket
2021/12/04 05:32:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:32:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:32:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:32:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:32:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:33:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:33:09 Waiting for dst bucket item count to get to: 5000 curr count: 3544
2021/12/04 05:33:14 Waiting for dst bucket item count to get to: 5000 curr count: 3544
2021/12/04 05:33:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:33:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:33:19 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/04 05:33:19 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/04 05:33:19 Pausing app: bucket_op_with_timer
2021/12/04 05:33:19 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:33:24 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/04 05:33:24 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/04 05:33:24 Resuming app: bucket_op_with_timer
2021/12/04 05:33:25 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:33:25 Starting bucket ops to source bucket
2021/12/04 05:33:33 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:33:38 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:33:43 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:33:48 Waiting for dst bucket item count to get to: 10000 curr count: 5254
2021/12/04 05:33:53 Waiting for dst bucket item count to get to: 10000 curr count: 5254
2021/12/04 05:33:58 Waiting for dst bucket item count to get to: 10000 curr count: 9797
2021/12/04 05:34:03 Waiting for dst bucket item count to get to: 10000 curr count: 9797
2021/12/04 05:34:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 05:34:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:34:08 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/04 05:34:08 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/04 05:34:08 Pausing app: bucket_op_with_timer
2021/12/04 05:34:08 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:34:13 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/04 05:34:13 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/04 05:34:13 Resuming app: bucket_op_with_timer
2021/12/04 05:34:14 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:34:14 Starting bucket ops to source bucket
2021/12/04 05:34:22 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 05:34:27 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 05:34:32 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 05:34:37 Waiting for dst bucket item count to get to: 15000 curr count: 10574
2021/12/04 05:34:42 Waiting for dst bucket item count to get to: 15000 curr count: 10574
2021/12/04 05:34:42 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/04 05:34:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:34:47 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/04 05:34:47 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/04 05:34:47 Pausing app: bucket_op_with_timer
2021/12/04 05:34:48 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:34:53 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/04 05:34:53 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/04 05:34:53 Resuming app: bucket_op_with_timer
2021/12/04 05:34:54 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:34:54 Starting bucket ops to source bucket
2021/12/04 05:35:02 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 05:35:07 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 05:35:12 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 05:35:17 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 05:35:22 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 05:35:27 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 05:35:27 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/04 05:35:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:35:32 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/04 05:35:32 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/04 05:35:32 Pausing app: bucket_op_with_timer
2021/12/04 05:35:32 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:35:37 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/04 05:35:37 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/04 05:35:37 Resuming app: bucket_op_with_timer
2021/12/04 05:35:38 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:35:38 Starting bucket ops to source bucket
2021/12/04 05:35:46 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 05:35:51 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 05:35:56 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 05:36:01 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 05:36:06 Waiting for dst bucket item count to get to: 25000 curr count: 20980
2021/12/04 05:36:11 Waiting for dst bucket item count to get to: 25000 curr count: 21774
2021/12/04 05:36:16 Waiting for dst bucket item count to get to: 25000 curr count: 23294
2021/12/04 05:36:16 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/04 05:36:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:36:21 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/04 05:36:21 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/04 05:36:21 Pausing app: bucket_op_with_timer
2021/12/04 05:36:21 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:36:26 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/04 05:36:26 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/04 05:36:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:36:31 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/04 05:36:36 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/04 05:36:41 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/04 05:36:46 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/04 05:36:46 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/04 05:36:46 No eventing-con process running
2021/12/04 05:36:46 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/04 05:36:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:36:58 Waiting for dst bucket item count to get to: 0 curr count: 18225
2021/12/04 05:37:03 Waiting for dst bucket item count to get to: 0 curr count: 5725
2021/12/04 05:37:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:37:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:37:15 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/04 05:37:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (309.28s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/04 05:37:25 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/04 05:37:25 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/04 05:37:25 No eventing-con process running
2021/12/04 05:37:25 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/04 05:37:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:37:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:37:34 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:37:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:37:39 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:37:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/04 05:37:45 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/04 05:37:50 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/04 05:37:55 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/04 05:38:00 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/04 05:38:00 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/04 05:38:00 Starting bucket ops to source bucket
2021/12/04 05:38:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:38:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:38:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:38:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:38:29 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/04 05:38:29 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:38:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:38:29 Pausing app: bucket_op_with_timer
2021/12/04 05:38:29 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:38:34 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/04 05:38:34 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/04 05:38:34 Resuming app: bucket_op_with_timer
2021/12/04 05:38:35 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:38:35 Starting bucket ops to source bucket
2021/12/04 05:38:48 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:38:53 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:38:58 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:39:03 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:39:08 Waiting for dst bucket item count to get to: 10000 curr count: 6708
2021/12/04 05:39:13 Waiting for dst bucket item count to get to: 10000 curr count: 6708
2021/12/04 05:39:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 05:39:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:39:18 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/04 05:39:23 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/04 05:39:28 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/04 05:39:33 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/04 05:39:33 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/04 05:39:33 No eventing-con process running
2021/12/04 05:39:33 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/04 05:39:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:39:45 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 05:39:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:39:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:39:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 05:39:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (161.39s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/04 05:40:06 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/04 05:40:06 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/04 05:40:06 No eventing-con process running
2021/12/04 05:40:06 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/04 05:40:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:40:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:40:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:40:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:40:20 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:40:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/04 05:40:26 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/04 05:40:31 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/04 05:40:36 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/04 05:40:41 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/04 05:40:41 App: TestPauseResumeWithEventingReb got deployed
2021/12/04 05:40:41 Starting bucket ops to source bucket
2021/12/04 05:40:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:40:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:41:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:41:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:41:11 Waiting for dst bucket item count to get to: 5000 curr count: 4991
2021/12/04 05:41:16 Waiting for dst bucket item count to get to: 5000 curr count: 4991
2021/12/04 05:41:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:41:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:41:16 Pausing app: bucket_op_with_timer
2021/12/04 05:41:16 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:41:21 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/04 05:41:21 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/04 05:41:21 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 05:41:21 Removing node(s): [] from the cluster
2021/12/04 05:41:26 Rebalance progress: 29.16666666666666
2021/12/04 05:41:31 Rebalance progress: 29.16666666666666
2021/12/04 05:41:36 Rebalance progress: 100
2021/12/04 05:41:36 Resuming app: bucket_op_with_timer
2021/12/04 05:41:38 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:41:38 Starting bucket ops to source bucket
2021/12/04 05:41:49 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:41:54 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:41:59 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:42:04 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 05:42:09 Waiting for dst bucket item count to get to: 10000 curr count: 5261
2021/12/04 05:42:14 Waiting for dst bucket item count to get to: 10000 curr count: 5261
2021/12/04 05:42:14 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 05:42:14 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 05:42:19 Rebalance progress: 29.16666666666666
2021/12/04 05:42:24 Rebalance progress: 29.16666666666666
2021/12/04 05:42:29 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/04 05:42:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:42:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:43:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:43:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:43:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:43:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:44:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:44:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:44:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:44:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:45:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:45:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:45:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 05:45:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 6527ac025c2c6435ecb86c340291986e
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
Producer node: 172.31.2.134:9300	Node UUID: 1c4be54b6cdadab18b15b27bfe63021d
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
Producer node: 127.0.0.1:9303	Node UUID: d2c9e4603ea95fc2ab44dbee3627c602
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/04 05:46:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:46:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:46:10 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/04 05:46:15 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/04 05:46:20 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/04 05:46:25 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/04 05:46:25 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/04 05:46:25 eventingSetup.go:46: No eventing-con process running
2021/12/04 05:46:25 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/04 05:46:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:46:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 05:46:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:46:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:46:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8602
2021/12/04 05:46:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3602
2021/12/04 05:46:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (417.16s)
=== RUN   TestBucketDeleteAfterPause
2021/12/04 05:47:04 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/04 05:47:04 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/04 05:47:04 eventingSetup.go:46: No eventing-con process running
2021/12/04 05:47:04 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/04 05:47:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:47:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:47:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:47:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:47:17 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:47:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/04 05:47:23 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/04 05:47:29 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/04 05:47:34 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/04 05:47:39 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/04 05:47:39 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/04 05:47:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 05:47:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 334
2021/12/04 05:47:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2834
2021/12/04 05:47:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:47:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:47:53 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/04 05:47:53 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:47:58 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/04 05:47:58 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/04 05:47:58 handler_test.go:716: Deleting source bucket: default
2021/12/04 05:48:05 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/04 05:48:05 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/04 05:48:10 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/04 05:48:10 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/04 05:48:10 eventingSetup.go:46: No eventing-con process running
2021/12/04 05:48:10 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/04 05:48:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:48:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:48:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:48:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3771
2021/12/04 05:48:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1271
2021/12/04 05:48:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (99.45s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/04 05:48:43 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/04 05:48:43 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/04 05:48:43 eventingSetup.go:46: No eventing-con process running
2021/12/04 05:48:43 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/04 05:48:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:48:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:48:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:48:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:48:56 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:48:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/04 05:49:02 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/04 05:49:07 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/04 05:49:12 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/04 05:49:17 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/04 05:49:17 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/04 05:49:17 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 05:49:52 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/04 05:49:53 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:49:58 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/04 05:49:58 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/04 05:52:58 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/04 05:52:58 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/04 05:52:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/04 05:53:04 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/04 05:53:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:53:04 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/04 05:53:09 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/04 05:53:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:53:09 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2829 expected 5000
2021/12/04 05:53:14 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/04 05:53:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:53:14 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3655 expected 5000
2021/12/04 05:53:19 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/04 05:53:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:53:19 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/04 05:53:19 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/04 05:53:19 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:53:29 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/04 05:53:34 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/04 05:53:39 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/04 05:53:39 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/04 05:53:39 eventingSetup.go:46: No eventing-con process running
2021/12/04 05:53:39 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/04 05:53:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:53:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4244
2021/12/04 05:53:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4244
2021/12/04 05:53:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:53:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:54:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (329.24s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/04 05:54:12 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/04 05:54:12 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/04 05:54:12 eventingSetup.go:46: No eventing-con process running
2021/12/04 05:54:12 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/04 05:54:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:54:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:54:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:54:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:54:26 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:54:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 05:54:32 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 05:54:37 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 05:54:42 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 05:54:47 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 05:54:47 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/04 05:54:47 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 05:54:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:55:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1901
2021/12/04 05:55:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1901
2021/12/04 05:55:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:55:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:55:06 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/04 05:55:06 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:55:36 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/04 05:55:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 05:55:42 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 05:55:47 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 05:55:52 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 05:55:57 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 05:55:57 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/04 05:55:57 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 05:56:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:56:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:56:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:56:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:56:15 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/04 05:56:15 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 05:56:25 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/04 05:56:30 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/04 05:56:35 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/04 05:56:35 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/04 05:56:35 eventingSetup.go:46: No eventing-con process running
2021/12/04 05:56:35 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/04 05:56:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:56:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:56:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:56:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (166.08s)
=== RUN   TestCPPWorkerCleanup
2021/12/04 05:56:58 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/04 05:56:58 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/04 05:56:58 eventingSetup.go:46: No eventing-con process running
2021/12/04 05:56:58 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/04 05:57:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:57:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:57:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:57:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:57:12 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:57:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/04 05:57:13 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 05:57:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:57:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:57:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:57:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 05:57:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2522
2021/12/04 05:57:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2522
2021/12/04 05:57:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 05:57:45 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 05:57:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 05:57:50 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/04 05:57:55 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/04 05:58:00 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/04 05:58:05 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/04 05:58:05 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/04 05:58:05 eventingSetup.go:46: No eventing-con process running
2021/12/04 05:58:05 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/04 05:58:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:58:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 05:58:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:58:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:58:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4204
2021/12/04 05:58:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1704
2021/12/04 05:58:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (134.98s)
=== RUN   TestWithUserXattrs
2021/12/04 05:59:13 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/04 05:59:13 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/04 05:59:13 eventingSetup.go:46: No eventing-con process running
2021/12/04 05:59:13 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/04 05:59:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:59:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:59:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 05:59:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 05:59:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 05:59:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 05:59:31 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/04 05:59:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/04 05:59:37 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/04 05:59:42 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/04 05:59:47 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/04 05:59:52 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/04 05:59:52 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/04 05:59:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 06:00:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:00:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:00:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 06:00:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:00:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:00:11 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/04 06:00:16 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/04 06:00:21 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/04 06:00:26 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/04 06:00:26 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/04 06:00:26 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:00:26 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/04 06:00:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:00:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2599
2021/12/04 06:00:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 99
2021/12/04 06:00:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:00:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:00:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 06:00:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.18s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/04 06:01:05 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 06:01:05 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/04 06:01:05 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:01:05 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/04 06:01:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:01:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:01:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:01:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:01:18 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:01:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/04 06:01:24 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/04 06:01:29 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/04 06:01:34 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/04 06:01:39 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/04 06:01:39 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/04 06:01:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 06:01:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:01:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 153
2021/12/04 06:01:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2653
2021/12/04 06:01:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 06:01:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:02:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:02:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3848
2021/12/04 06:02:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3848
2021/12/04 06:02:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:02:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:02:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4160
2021/12/04 06:02:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1660
2021/12/04 06:02:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:02:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 06:02:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 11677
2021/12/04 06:03:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 17877
2021/12/04 06:03:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 22110
2021/12/04 06:03:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/04 06:03:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:03:09 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 06:03:14 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 06:03:19 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 06:03:24 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 06:03:29 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 06:03:29 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/04 06:03:29 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:03:34 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 06:03:34 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/04 06:03:34 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:03:34 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/04 06:03:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:03:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/04 06:03:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:03:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:03:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/04 06:03:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (182.82s)
=== RUN   TestMetaBucketDelete
2021/12/04 06:04:07 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/04 06:04:07 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/04 06:04:07 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:04:07 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/04 06:04:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:04:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:04:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:04:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:04:21 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:04:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 06:04:27 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/04 06:04:32 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/04 06:04:37 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/04 06:04:42 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/04 06:04:42 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/04 06:04:42 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/04 06:04:43 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/04 06:04:48 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/04 06:04:48 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/04 06:05:03 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/04 06:05:03 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/04 06:05:03 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:05:03 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/04 06:05:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:05:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:05:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:05:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (79.20s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/04 06:05:27 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 06:05:27 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 06:05:27 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:05:27 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/04 06:05:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:05:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:05:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:05:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:05:40 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:05:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/04 06:06:00 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/04 06:06:00 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/04 06:06:15 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 06:06:15 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 06:06:30 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 06:06:30 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 06:06:30 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:06:30 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/04 06:06:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:06:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:06:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:06:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.96s)
=== RUN   TestSourceBucketDelete
2021/12/04 06:06:54 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/04 06:06:54 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/04 06:06:54 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:06:54 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/04 06:06:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:07:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:07:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:07:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:07:07 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:07:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 06:07:13 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/04 06:07:18 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/04 06:07:23 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/04 06:07:28 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/04 06:07:28 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/04 06:07:28 handler_test.go:945: Deleting source bucket: default
2021/12/04 06:07:29 handler_test.go:947: Deleted source bucket: default
2021/12/04 06:07:34 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/04 06:07:34 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/04 06:07:50 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/04 06:07:50 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/04 06:07:50 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:07:50 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/04 06:07:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:07:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:07:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:08:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.26s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/04 06:08:13 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/04 06:08:13 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/04 06:08:13 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:08:13 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/04 06:08:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:08:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:08:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:08:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:08:26 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:08:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/04 06:08:46 handler_test.go:966: Deleting source bucket: default
2021/12/04 06:08:47 handler_test.go:968: Deleted source bucket: default
2021/12/04 06:09:02 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/04 06:09:02 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/04 06:09:17 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/04 06:09:17 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/04 06:09:17 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:09:18 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/04 06:09:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:09:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:09:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:09:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.80s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/04 06:09:41 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 06:09:41 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 06:09:41 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:09:41 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/04 06:09:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:09:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:09:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:09:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:09:54 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:09:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/04 06:10:14 handler_test.go:988: Deleting source bucket: default
2021/12/04 06:10:15 handler_test.go:990: Deleted source bucket: default
2021/12/04 06:10:15 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/04 06:10:16 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/04 06:10:31 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 06:10:31 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 06:10:46 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 06:10:46 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 06:10:46 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:10:46 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/04 06:10:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:10:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:10:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:10:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.31s)
=== RUN   TestUndeployDuringBootstrap
2021/12/04 06:11:09 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/04 06:11:09 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/04 06:11:09 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:11:09 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/04 06:11:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:11:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:11:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:11:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:11:22 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:11:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/04 06:11:23 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 06:11:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:11:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:11:35 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/04 06:11:35 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/04 06:11:35 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/04 06:11:40 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/04 06:11:45 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/04 06:11:50 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/04 06:11:55 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/04 06:12:00 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/04 06:12:00 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/04 06:12:00 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:12:25 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/04 06:12:25 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/04 06:12:25 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:12:26 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/04 06:12:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:12:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 06:12:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:12:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:12:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4490
2021/12/04 06:12:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1990
2021/12/04 06:12:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (114.76s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/04 06:12:59 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:13:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/04 06:13:05 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/04 06:13:10 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/04 06:13:15 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/04 06:13:20 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/04 06:13:20 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/04 06:13:20 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/04 06:13:20 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 06:13:20 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/04 06:13:25 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/04 06:13:25 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/04 06:13:25 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:13:25 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/04 06:13:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:13:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:13:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:13:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.67s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/04 06:13:48 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 06:13:48 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/04 06:13:48 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:13:48 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/04 06:13:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:13:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:13:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:14:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:14:02 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:14:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/04 06:14:08 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/04 06:14:13 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/04 06:14:18 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/04 06:14:23 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/04 06:14:23 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/04 06:14:23 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 06:14:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:14:58 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 06:15:03 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 06:15:08 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 06:15:13 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 06:15:13 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/04 06:15:13 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:16:58 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 06:16:58 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/04 06:16:58 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:16:58 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/04 06:17:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:17:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 06:17:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 06:17:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:17:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:17:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 06:17:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.13s)
=== RUN   TestUndeployWithKVFailover
2021/12/04 06:17:36 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/04 06:17:36 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/04 06:17:36 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:17:36 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/04 06:17:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:17:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:17:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:17:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:17:50 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:17:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/04 06:17:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 06:17:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:18:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:18:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:18:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:18:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:18:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:18:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2159
2021/12/04 06:18:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2159
2021/12/04 06:18:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 06:18:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 06:18:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 06:18:38 Rebalance progress: 29.16666666666666
2021/12/04 06:18:43 Rebalance progress: 29.16666666666666
2021/12/04 06:18:48 Rebalance progress: 100
2021/12/04 06:18:48 Failing over: [127.0.0.1:19003]
2021/12/04 06:18:58 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 06:19:58 Removing node(s): [] from the cluster
2021/12/04 06:20:03 Rebalance progress: 29.16666666666666
2021/12/04 06:20:08 Rebalance progress: 29.16666666666666
2021/12/04 06:20:13 Rebalance progress: 100
2021/12/04 06:20:18 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/04 06:20:18 App: TestUndeployWithKVFailover got un-deployed
2021/12/04 06:20:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:20:23 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/04 06:20:23 App: TestUndeployWithKVFailover got un-deployed
2021/12/04 06:20:23 No eventing-con process running
2021/12/04 06:20:23 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/04 06:20:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:20:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 06:20:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:20:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:20:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 06:20:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (200.04s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/04 06:20:56 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/04 06:20:56 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/04 06:20:57 No eventing-con process running
2021/12/04 06:20:57 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/04 06:20:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:21:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:21:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:21:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:21:10 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:21:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/04 06:21:16 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/04 06:21:21 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/04 06:21:26 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/04 06:21:31 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/04 06:21:31 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/04 06:21:31 Starting bucket ops to source bucket
2021/12/04 06:21:52 Waiting for dst bucket item count to get to: 20000 curr count: 9416
2021/12/04 06:21:52 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/04 06:21:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:21:54 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:22:04 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/04 06:22:09 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 06:22:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:22:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:22:21 Waiting for dst bucket item count to get to: 0 curr count: 18040
2021/12/04 06:22:26 Waiting for dst bucket item count to get to: 0 curr count: 18040
2021/12/04 06:22:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:22:26 Starting bucket ops to source bucket
2021/12/04 06:22:39 Waiting for dst bucket item count to get to: 10000 curr count: 2910
2021/12/04 06:22:44 Waiting for dst bucket item count to get to: 10000 curr count: 2910
2021/12/04 06:22:44 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 06:22:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:22:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:22:49 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/04 06:22:54 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/04 06:22:59 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/04 06:23:04 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/04 06:23:04 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/04 06:23:04 No eventing-con process running
2021/12/04 06:23:04 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/04 06:23:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:23:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 06:23:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:23:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:23:27 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 06:23:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (160.62s)
=== RUN   TestTimerOverwrite
2021/12/04 06:23:37 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/04 06:23:37 App: TestTimerOverwrite got un-deployed
2021/12/04 06:23:37 No eventing-con process running
2021/12/04 06:23:37 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/04 06:23:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:23:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:23:45 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:23:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:23:50 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:23:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/04 06:23:56 Waiting for app: TestTimerOverwrite to get deployed
2021/12/04 06:24:01 Waiting for app: TestTimerOverwrite to get deployed
2021/12/04 06:24:07 Waiting for app: TestTimerOverwrite to get deployed
2021/12/04 06:24:12 Waiting for app: TestTimerOverwrite to get deployed
2021/12/04 06:24:12 App: TestTimerOverwrite got deployed
2021/12/04 06:24:22 Starting bucket ops to source bucket
2021/12/04 06:24:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:24:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:24:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:24:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:24:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:24:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:25:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:25:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:25:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:25:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:25:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:25:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:25:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 06:25:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:25:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:25:41 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/04 06:25:46 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/04 06:25:51 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/04 06:25:56 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/04 06:25:56 App: TestTimerOverwrite got un-deployed
2021/12/04 06:25:56 No eventing-con process running
2021/12/04 06:25:56 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/04 06:25:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:26:08 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 06:26:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:26:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:26:19 Waiting for dst bucket item count to get to: 0 curr count: 2529
2021/12/04 06:26:24 Waiting for dst bucket item count to get to: 0 curr count: 2529
2021/12/04 06:26:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (177.40s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/04 06:26:30 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 06:26:30 Removing node(s): [] from the cluster
2021/12/04 06:26:35 Rebalance progress: 100
2021/12/04 06:26:35 Failing over: [127.0.0.1:19003]
2021/12/04 06:26:40 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/04 06:26:40 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/04 06:26:40 No eventing-con process running
2021/12/04 06:26:40 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/04 06:26:42 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:26:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:26:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:26:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:26:53 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:26:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/04 06:26:55 Starting bucket ops to source bucket
2021/12/04 06:27:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:27:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:27:05 Context: Post to set retry counter, response code: 200 dump: 
2021/12/04 06:27:05 set retry counter, response &{[] }
2021/12/04 06:28:10 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/04 06:28:10 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/04 06:28:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:28:10 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/04 06:28:15 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/04 06:28:15 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/04 06:28:15 No eventing-con process running
2021/12/04 06:28:15 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/04 06:28:16 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:28:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 06:28:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:28:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:28:38 Waiting for dst bucket item count to get to: 0 curr count: 2977
2021/12/04 06:28:43 Waiting for dst bucket item count to get to: 0 curr count: 477
2021/12/04 06:28:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:28:43 Removing node(s): [] from the cluster
2021/12/04 06:28:48 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (143.33s)
=== RUN   TestOnUpdateSrcMutation
2021/12/04 06:28:58 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/04 06:28:58 App: TestOnUpdateSrcMutation got un-deployed
2021/12/04 06:28:58 No eventing-con process running
2021/12/04 06:28:58 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/04 06:28:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:29:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:29:06 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:29:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:29:11 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:29:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/04 06:29:12 Starting bucket ops to source bucket
2021/12/04 06:29:34 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 06:29:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:29:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:29:39 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/04 06:29:44 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/04 06:29:49 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/04 06:29:54 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/04 06:29:54 App: TestOnUpdateSrcMutation got un-deployed
2021/12/04 06:29:54 No eventing-con process running
2021/12/04 06:29:54 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/04 06:29:56 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:30:06 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 06:30:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:30:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:30:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (84.58s)
=== RUN   TestOnDeleteSrcMutation
2021/12/04 06:30:22 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/04 06:30:22 App: TestOnDeleteSrcMutation got un-deployed
2021/12/04 06:30:22 No eventing-con process running
2021/12/04 06:30:22 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/04 06:30:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:30:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:30:31 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:30:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:30:36 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:30:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/04 06:30:37 Starting bucket ops to source bucket
2021/12/04 06:31:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 06:31:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:31:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:31:11 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/04 06:31:16 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/04 06:31:21 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/04 06:31:26 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/04 06:31:26 App: TestOnDeleteSrcMutation got un-deployed
2021/12/04 06:31:26 No eventing-con process running
2021/12/04 06:31:26 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/04 06:31:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:31:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 06:31:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:31:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:31:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (91.37s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/04 06:31:54 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/04 06:31:54 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/04 06:31:54 No eventing-con process running
2021/12/04 06:31:54 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/04 06:31:55 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:32:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:32:02 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:32:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:32:07 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:32:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 06:32:08 Starting bucket ops to source bucket
2021/12/04 06:32:45 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 06:32:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:32:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:32:50 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/04 06:32:55 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/04 06:33:00 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/04 06:33:05 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/04 06:33:05 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/04 06:33:05 No eventing-con process running
2021/12/04 06:33:05 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/04 06:33:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:33:17 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 06:33:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:33:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:33:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.71s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/04 06:33:34 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/04 06:33:34 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/04 06:33:34 No eventing-con process running
2021/12/04 06:33:34 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/04 06:33:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:33:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:33:42 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:33:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:33:47 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:33:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 06:33:48 Starting bucket ops to source bucket
2021/12/04 06:34:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 06:34:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:34:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:34:32 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/04 06:34:37 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/04 06:34:42 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/04 06:34:47 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/04 06:34:47 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/04 06:34:47 No eventing-con process running
2021/12/04 06:34:47 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/04 06:34:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:34:58 Waiting for dst bucket item count to get to: 0 curr count: 4096
2021/12/04 06:35:03 Waiting for dst bucket item count to get to: 0 curr count: 1596
2021/12/04 06:35:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:35:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:35:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (106.59s)
=== RUN   TestInterHandlerRecursion
2021/12/04 06:35:20 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 06:35:20 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/04 06:35:20 No eventing-con process running
2021/12/04 06:35:20 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/04 06:35:21 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:35:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:35:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:35:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:35:38 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 06:35:38 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/04 06:35:38 No eventing-con process running
2021/12/04 06:35:38 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/04 06:35:40 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:35:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:35:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:35:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:35:51 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:35:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/04 06:35:53 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/04 06:35:58 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 06:36:03 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 06:36:08 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 06:36:13 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 06:36:13 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/04 06:36:13 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:36:13 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/04 06:36:13 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/04 06:36:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:36:18 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 06:36:23 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 06:36:28 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 06:36:33 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 06:36:33 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/04 06:36:33 No eventing-con process running
2021/12/04 06:36:33 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/04 06:36:34 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:36:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:36:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:36:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:36:51 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 06:36:51 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/04 06:36:51 No eventing-con process running
2021/12/04 06:36:51 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/04 06:36:52 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:36:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:36:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:37:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (114.18s)
=== RUN   TestInterBucketRecursion
2021/12/04 06:37:09 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:37:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/04 06:37:10 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/04 06:37:10 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:37:10 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/04 06:37:15 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/04 06:37:20 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/04 06:37:25 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/04 06:37:30 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/04 06:37:30 App: TestInterBucketRecursion_function1 got deployed
2021/12/04 06:37:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:37:36 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/04 06:37:41 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/04 06:37:46 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/04 06:37:51 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/04 06:37:51 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/04 06:37:51 No eventing-con process running
2021/12/04 06:37:51 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/04 06:37:52 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:37:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:37:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:38:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:38:09 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/04 06:38:09 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/04 06:38:09 No eventing-con process running
2021/12/04 06:38:09 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/04 06:38:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:38:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:38:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:38:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.69s)
=== RUN   TestLargeHandler
2021/12/04 06:38:22 Changed force_compress value to false
2021/12/04 06:38:22 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:38:22 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/04 06:38:22 Changed force_compress value to true
2021/12/04 06:38:22 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:38:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/04 06:38:29 Waiting for app: TestLargeHandler_function to get deployed
2021/12/04 06:38:34 Waiting for app: TestLargeHandler_function to get deployed
2021/12/04 06:38:39 Waiting for app: TestLargeHandler_function to get deployed
2021/12/04 06:38:44 Waiting for app: TestLargeHandler_function to get deployed
2021/12/04 06:38:44 App: TestLargeHandler_function got deployed
2021/12/04 06:38:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:38:49 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/04 06:38:54 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/04 06:38:59 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/04 06:38:59 App: TestLargeHandler_function got un-deployed
2021/12/04 06:38:59 No eventing-con process running
2021/12/04 06:38:59 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/04 06:39:01 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:39:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:39:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:39:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (50.40s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/04 06:39:22 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 06:39:22 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/04 06:39:22 No eventing-con process running
2021/12/04 06:39:22 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/04 06:39:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:39:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:39:31 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:39:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:39:41 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 06:39:41 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/04 06:39:41 No eventing-con process running
2021/12/04 06:39:41 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/04 06:39:42 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:39:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:39:49 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:39:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:39:54 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:39:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/04 06:39:55 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/04 06:40:00 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 06:40:05 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 06:40:10 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 06:40:15 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 06:40:15 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/04 06:40:15 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:40:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/04 06:40:16 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/04 06:40:21 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/04 06:40:26 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/04 06:40:31 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/04 06:40:36 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/04 06:40:36 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/04 06:40:36 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/04 06:40:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:40:41 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 06:40:46 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 06:40:51 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 06:40:56 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 06:40:56 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/04 06:40:56 No eventing-con process running
2021/12/04 06:40:56 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/04 06:40:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:41:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:41:06 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:41:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:41:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:41:16 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 06:41:21 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 06:41:26 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 06:41:31 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 06:41:31 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/04 06:41:31 No eventing-con process running
2021/12/04 06:41:31 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/04 06:41:32 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:41:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:41:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:41:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.78s)
=== RUN   TestAllowInterBucketRecursion
2021/12/04 06:41:49 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:41:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/04 06:41:50 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/04 06:41:55 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/04 06:42:00 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/04 06:42:05 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/04 06:42:10 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/04 06:42:10 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/04 06:42:10 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:42:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/04 06:42:11 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/04 06:42:16 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/04 06:42:21 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/04 06:42:27 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/04 06:42:32 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/04 06:42:32 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/04 06:42:32 TestAllowInterBucketRecursion_function2 is deployed
2021/12/04 06:42:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:42:37 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/04 06:42:42 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/04 06:42:47 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/04 06:42:52 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/04 06:42:52 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/04 06:42:52 No eventing-con process running
2021/12/04 06:42:52 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/04 06:42:53 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:42:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:43:01 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:43:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:43:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:43:11 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/04 06:43:16 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/04 06:43:21 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/04 06:43:26 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/04 06:43:26 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/04 06:43:26 No eventing-con process running
2021/12/04 06:43:26 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/04 06:43:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:43:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:43:34 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:43:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.24s)
=== RUN   TestInterBucketRecursion2
2021/12/04 06:43:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:43:52 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:43:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:44:04 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:44:09 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:44:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/04 06:44:15 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/04 06:44:20 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/04 06:44:25 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/04 06:44:30 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/04 06:44:30 App: TestInterBucketRecursion2_function1 got deployed
2021/12/04 06:44:30 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:44:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/04 06:44:36 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/04 06:44:41 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/04 06:44:46 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/04 06:44:51 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/04 06:44:51 App: TestInterBucketRecursion2_function2 got deployed
2021/12/04 06:44:51 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:44:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/04 06:44:57 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/04 06:45:02 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/04 06:45:07 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/04 06:45:12 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/04 06:45:12 App: TestInterBucketRecursion2_function3 got deployed
2021/12/04 06:45:12 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:45:12 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/04 06:45:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:45:17 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/04 06:45:22 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/04 06:45:27 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/04 06:45:32 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/04 06:45:32 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/04 06:45:32 No eventing-con process running
2021/12/04 06:45:32 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/04 06:45:34 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:45:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:45:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:45:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:45:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:45:51 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/04 06:45:56 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/04 06:46:01 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/04 06:46:06 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/04 06:46:06 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/04 06:46:06 No eventing-con process running
2021/12/04 06:46:06 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/04 06:46:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:46:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:46:14 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:46:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:46:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:46:24 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/04 06:46:30 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/04 06:46:35 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/04 06:46:40 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/04 06:46:40 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/04 06:46:40 No eventing-con process running
2021/12/04 06:46:40 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/04 06:46:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:46:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:46:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:46:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:46:58 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/04 06:46:58 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/04 06:46:58 No eventing-con process running
2021/12/04 06:46:58 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/04 06:46:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:47:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:47:06 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:47:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.62s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/04 06:47:18 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/04 06:47:23 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/04 06:47:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 06:47:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 06:47:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 06:48:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 06:48:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 06:48:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 06:48:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 06:49:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 06:49:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 06:49:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 06:49:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/04 06:50:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:50:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 06:50:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 06:50:17 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/04 06:50:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:50:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 06:50:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 06:50:23 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/04 06:50:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:50:28 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/04 06:50:28 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/04 06:50:28 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:50:28 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/04 06:50:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:50:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:50:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:50:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:50:46 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:50:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 06:50:52 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/04 06:50:57 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/04 06:51:02 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/04 06:51:07 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/04 06:51:07 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 6527ac025c2c6435ecb86c340291986e
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 1c4be54b6cdadab18b15b27bfe63021d
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: d2c9e4603ea95fc2ab44dbee3627c602
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]


2021/12/04 06:51:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:51:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 06:51:12 Rebalance progress: 29.16666666666666
2021/12/04 06:51:17 Rebalance progress: 29.16666666666666
2021/12/04 06:51:22 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 6527ac025c2c6435ecb86c340291986e
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
Producer node: 172.31.2.134:9300	Node UUID: 1c4be54b6cdadab18b15b27bfe63021d
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: d2c9e4603ea95fc2ab44dbee3627c602
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]


2021/12/04 06:51:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:51:22 handler_test.go:1674: Deleting source bucket: default
2021/12/04 06:51:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 06:51:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 06:51:29 Rebalance progress: 29.16666666666666
2021/12/04 06:51:34 Rebalance progress: 29.16666666666666
2021/12/04 06:51:39 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 900
	vb list dump: [0-29, 31, 33-49, 51-56, 58-59, 61-65, 67-80, 82-83, 87, 89-91, 93-98, 100, 102-125, 127-136, 138-144, 146-147, 149-172, 174, 176, 178-181, 183-196, 198-203, 205-208, 210, 212, 214-218, 220-239, 244-245, 247-248, 250-258, 260-262, 265-268, 270-281, 283-304, 306, 308-309, 311-319, 321-333, 335-351, 353, 355-370, 372-380, 382-383, 385-395, 397-400, 402-406, 408-411, 413-421, 423-427, 429, 431-452, 454-455, 457-462, 464-467, 469-474, 477, 480-486, 488-491, 493, 495, 497-502, 504, 506-580, 582-586, 588-589, 591, 593-611, 613-616, 618-639, 641-653, 656-659, 661-667, 670-677, 679-691, 694-701, 703-704, 706-712, 714-720, 722-726, 728-729, 731-732, 734-742, 744-751, 753-775, 777-785, 787-788, 790-797, 799-801, 804-807, 809, 811-825, 827-830, 832-833, 835-836, 838-860, 864, 867-888, 890-926, 928, 930-939, 941, 943-944, 947-961, 963-966, 968-977, 979-985, 987-989, 991-999, 1001-1023]
	stream status: stopped
	len: 124
	vb list dump: [30, 32, 50, 57, 60, 66, 81, 84-86, 88, 92, 99, 101, 126, 137, 145, 148, 173, 175, 177, 182, 197, 204, 209, 211, 213, 219, 240-243, 246, 249, 259, 263-264, 269, 282, 305, 307, 310, 320, 334, 352, 354, 371, 381, 384, 396, 401, 407, 412, 422, 428, 430, 453, 456, 463, 468, 475-476, 478-479, 487, 492, 494, 496, 503, 505, 581, 587, 590, 592, 612, 617, 640, 654-655, 660, 668-669, 678, 692-693, 702, 705, 713, 721, 727, 730, 733, 743, 752, 776, 786, 789, 798, 802-803, 808, 810, 826, 831, 834, 837, 861-863, 865-866, 889, 927, 929, 940, 942, 945-946, 962, 967, 978, 986, 990, 1000]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: 1c4be54b6cdadab18b15b27bfe63021d
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-926, 928, 930-939, 941, 943-944, 947-961, 963-966, 968-977, 979-985, 987-989, 991-999, 1001-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [683-691, 694-701, 703-704, 706-712, 714-720, 722-726, 728-729, 731-732, 734-742, 744-751, 753-775, 777-785, 787-788, 790-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [797, 799-801, 804-807, 809, 811-825, 827-830, 832-833, 835-836, 838-860, 864, 867-888, 890-910]
Producer node: 127.0.0.1:9303	Node UUID: d2c9e4603ea95fc2ab44dbee3627c602
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [342-351, 353, 355-370, 372-380, 382-383, 385-395, 397-400, 402-406, 408-411, 413-421, 423-427, 429, 431-452, 454-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [457-462, 464-467, 469-474, 477, 480-486, 488-491, 493, 495, 497-502, 504, 506-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570-580, 582-586, 588-589, 591, 593-611, 613-616, 618-639, 641-653, 656-659, 661-667, 670-677, 679-682]
Producer node: 127.0.0.1:9302	Node UUID: 6527ac025c2c6435ecb86c340291986e
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [0-29, 31, 33-49, 51-56, 58-59, 61-65, 67-80, 82-83, 87, 89-91, 93-98, 100, 102-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [114-125, 127-136, 138-144, 146-147, 149-172, 174, 176, 178-181, 183-196, 198-203, 205-208, 210, 212, 214-218, 220-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 98
	v: [228-239, 244-245, 247-248, 250-258, 260-262, 265-268, 270-281, 283-304, 306, 308-309, 311-319, 321-333, 335-341]


2021/12/04 06:51:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 06:51:45 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/04 06:51:45 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/04 06:51:50 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/04 06:51:50 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/04 06:51:50 eventingSetup.go:46: No eventing-con process running
2021/12/04 06:51:50 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/04 06:51:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:51:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:51:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:52:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:52:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/04 06:52:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 06:52:08 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (298.17s)
=== RUN   TestN1QLRecursion
2021/12/04 06:52:11 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 06:52:12 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/04 06:52:12 Removing node(s): [] from the cluster
2021/12/04 06:52:17 Rebalance progress: 100
2021/12/04 06:52:22 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:52:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:52:34 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:52:39 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:52:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/04 06:52:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/04 06:52:45 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/04 06:52:50 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/04 06:52:55 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/04 06:53:01 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/04 06:53:01 App: TestN1QLRecursion_function1 got deployed
2021/12/04 06:53:01 TestN1QLRecursion_function1 is deployed
2021/12/04 06:53:01 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:53:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/04 06:53:02 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/04 06:53:07 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/04 06:53:12 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/04 06:53:17 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/04 06:53:22 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/04 06:53:22 App: TestN1QLRecursion_function3 got deployed
2021/12/04 06:53:22 TestN1QLRecursion_function3 is deployed
2021/12/04 06:53:22 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:53:22 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/04 06:53:22 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/04 06:53:22 Success: TestN1QLRecursion_function2 is not deployed
2021/12/04 06:53:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:53:27 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/04 06:53:32 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/04 06:53:37 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/04 06:53:42 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/04 06:53:42 App: TestN1QLRecursion_function1 got un-deployed
2021/12/04 06:53:42 No eventing-con process running
2021/12/04 06:53:42 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/04 06:53:44 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:53:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:53:51 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:53:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:53:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:54:01 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/04 06:54:06 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/04 06:54:11 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/04 06:54:16 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/04 06:54:16 App: TestN1QLRecursion_function3 got un-deployed
2021/12/04 06:54:16 No eventing-con process running
2021/12/04 06:54:16 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/04 06:54:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:54:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:54:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:54:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:54:31 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 06:54:36 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.13s)
=== RUN   TestN1QLAllowRecursion
2021/12/04 06:54:36 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 06:54:36 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/04 06:54:36 Removing node(s): [] from the cluster
2021/12/04 06:54:42 Rebalance progress: 100
2021/12/04 06:54:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:54:54 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:54:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:55:04 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:55:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/04 06:55:05 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/04 06:55:10 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/04 06:55:15 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/04 06:55:20 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/04 06:55:25 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/04 06:55:26 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/04 06:55:26 TestN1QLAllowRecursion_function1 is deployed
2021/12/04 06:55:26 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:55:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/04 06:55:27 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/04 06:55:32 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/04 06:55:37 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/04 06:55:42 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/04 06:55:47 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/04 06:55:47 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/04 06:55:47 TestN1QLAllowRecursion_function3 is deployed
2021/12/04 06:55:47 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:55:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/04 06:55:48 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/04 06:55:53 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/04 06:55:58 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/04 06:56:03 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/04 06:56:08 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/04 06:56:08 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/04 06:56:08 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/04 06:56:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:56:13 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/04 06:56:18 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/04 06:56:23 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/04 06:56:28 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/04 06:56:28 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/04 06:56:28 No eventing-con process running
2021/12/04 06:56:28 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/04 06:56:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:56:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:56:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:56:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:56:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:56:47 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/04 06:56:52 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/04 06:56:57 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/04 06:57:02 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/04 06:57:02 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/04 06:57:02 No eventing-con process running
2021/12/04 06:57:02 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/04 06:57:04 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:57:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:57:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:57:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:57:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 06:57:21 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/04 06:57:26 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/04 06:57:31 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/04 06:57:36 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/04 06:57:36 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/04 06:57:36 No eventing-con process running
2021/12/04 06:57:36 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/04 06:57:38 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:57:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:57:45 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:57:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:57:52 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 06:57:57 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.42s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/04 06:57:57 Sleeping for some time
2021/12/04 06:58:07 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/04 06:58:07 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/04 06:58:07 No eventing-con process running
2021/12/04 06:58:07 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/04 06:58:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:58:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:58:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 06:58:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 06:58:20 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 06:58:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 06:58:22 Starting bucket ops to source bucket
2021/12/04 06:59:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:59:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 06:59:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 06:59:16 Starting bucket ops to source bucket
2021/12/04 06:59:17 Starting bucket ops to source bucket
2021/12/04 07:00:23 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/04 07:00:28 Waiting for dst bucket item count to get to: 7000 curr count: 6000
2021/12/04 07:00:28 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/04 07:00:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:00:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:00:33 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/04 07:00:38 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/04 07:00:43 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/04 07:00:48 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/04 07:00:48 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/04 07:00:48 No eventing-con process running
2021/12/04 07:00:48 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/04 07:00:50 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:01:00 Waiting for dst bucket item count to get to: 0 curr count: 610
2021/12/04 07:01:05 Waiting for dst bucket item count to get to: 0 curr count: 610
2021/12/04 07:01:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:01:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:01:17 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/04 07:01:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (200.15s)
=== RUN   TestTimerOverWriteSameReference
2021/12/04 07:01:22 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/04 07:01:22 App: TestTimerOverWriteSameReference got un-deployed
2021/12/04 07:01:22 No eventing-con process running
2021/12/04 07:01:22 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/04 07:01:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:01:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:01:31 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:01:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:01:36 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:01:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/04 07:01:42 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/04 07:01:47 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/04 07:01:52 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/04 07:01:57 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/04 07:01:57 App: TestTimerOverWriteSameReference got deployed
2021/12/04 07:02:07 Starting bucket ops to source bucket
2021/12/04 07:03:08 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/04 07:03:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:03:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:03:23 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/04 07:03:28 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/04 07:03:33 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/04 07:03:38 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/04 07:03:38 App: TestTimerOverWriteSameReference got un-deployed
2021/12/04 07:03:38 No eventing-con process running
2021/12/04 07:03:38 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/04 07:03:40 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:03:50 Waiting for dst bucket item count to get to: 0 curr count: 662
2021/12/04 07:03:55 Waiting for dst bucket item count to get to: 0 curr count: 662
2021/12/04 07:03:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:03:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:04:07 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/04 07:04:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (170.15s)
=== RUN   TestCancelTimerBucketop
2021/12/04 07:04:12 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/04 07:04:12 App: TestCancelTimerBucketop got un-deployed
2021/12/04 07:04:12 No eventing-con process running
2021/12/04 07:04:12 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/04 07:04:14 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:04:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:04:21 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:04:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:04:26 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:04:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/04 07:04:32 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/04 07:04:37 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/04 07:04:42 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/04 07:04:47 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/04 07:04:47 App: TestCancelTimerBucketop got deployed
2021/12/04 07:04:47 Starting bucket ops to source bucket
2021/12/04 07:04:54 Starting bucket ops to source bucket
2021/12/04 07:05:56 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/04 07:05:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:05:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:06:02 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/04 07:06:07 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/04 07:06:12 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/04 07:06:17 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/04 07:06:17 App: TestCancelTimerBucketop got un-deployed
2021/12/04 07:06:17 No eventing-con process running
2021/12/04 07:06:17 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/04 07:06:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:06:29 Waiting for dst bucket item count to get to: 0 curr count: 813
2021/12/04 07:06:34 Waiting for dst bucket item count to get to: 0 curr count: 313
2021/12/04 07:06:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:06:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:06:45 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 07:06:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.47s)
=== RUN   TestJSExpiryDate
2021/12/04 07:06:50 Starting bucket ops to source bucket
2021/12/04 07:07:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 07:07:04 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/04 07:07:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/04 07:07:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:07:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:07:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:07:40 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/04 07:07:45 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/04 07:07:50 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/04 07:07:55 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/04 07:07:55 App: TestJSExpiryDate got un-deployed
2021/12/04 07:07:55 No eventing-con process running
2021/12/04 07:07:55 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/04 07:07:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:08:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:08:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:08:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (84.03s)
=== RUN   TestBinaryDoc
2021/12/04 07:08:15 Starting bucket ops to source bucket
2021/12/04 07:08:15 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:08:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/04 07:08:16 Starting bucket ops to source bucket
2021/12/04 07:08:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:08:16 Deleting source bucket docs now
2021/12/04 07:08:16 Starting bucket ops to source bucket
2021/12/04 07:08:53 Waiting for dst bucket item count to get to: 32 curr count: 21
2021/12/04 07:08:58 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/04 07:09:03 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/04 07:09:03 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/04 07:09:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:09:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:09:08 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/04 07:09:13 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/04 07:09:18 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/04 07:09:23 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/04 07:09:23 App: TestBinaryDoc got un-deployed
2021/12/04 07:09:23 No eventing-con process running
2021/12/04 07:09:23 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/04 07:09:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:09:35 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/04 07:09:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:09:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:09:47 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/04 07:09:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.53s)
=== RUN   TestConstantBindings
2021/12/04 07:09:47 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:09:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/04 07:09:48 Starting bucket ops to source bucket
2021/12/04 07:09:54 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/04 07:09:59 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/04 07:10:04 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/04 07:10:09 Waiting for dst bucket item count to get to: 6000 curr count: 3
2021/12/04 07:10:14 Waiting for dst bucket item count to get to: 6000 curr count: 3003
2021/12/04 07:10:14 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/04 07:10:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:10:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:10:19 Waiting for app: TestConstantBindings to get un-deployed
2021/12/04 07:10:24 Waiting for app: TestConstantBindings to get un-deployed
2021/12/04 07:10:29 Waiting for app: TestConstantBindings to get un-deployed
2021/12/04 07:10:34 Waiting for app: TestConstantBindings to get un-deployed
2021/12/04 07:10:34 App: TestConstantBindings got un-deployed
2021/12/04 07:10:34 No eventing-con process running
2021/12/04 07:10:34 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/04 07:10:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:10:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 07:10:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:10:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:10:58 Waiting for dst bucket item count to get to: 0 curr count: 3243
2021/12/04 07:11:03 Waiting for dst bucket item count to get to: 0 curr count: 243
2021/12/04 07:11:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (75.67s)
=== RUN   TestCodeInsight
2021/12/04 07:11:08 Waiting for app: TestCodeInsight to get un-deployed
2021/12/04 07:11:08 App: TestCodeInsight got un-deployed
2021/12/04 07:11:08 No eventing-con process running
2021/12/04 07:11:08 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/04 07:11:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:11:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:11:16 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:11:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:11:21 Starting bucket ops to source bucket
2021/12/04 07:11:22 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:11:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/04 07:11:28 Waiting for app: TestCodeInsight to get deployed
2021/12/04 07:11:33 Waiting for app: TestCodeInsight to get deployed
2021/12/04 07:11:38 Waiting for app: TestCodeInsight to get deployed
2021/12/04 07:11:43 Waiting for app: TestCodeInsight to get deployed
2021/12/04 07:11:43 App: TestCodeInsight got deployed
2021/12/04 07:11:43 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/04 07:11:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:11:48 Waiting for app: TestCodeInsight to get un-deployed
2021/12/04 07:11:53 Waiting for app: TestCodeInsight to get un-deployed
2021/12/04 07:11:58 Waiting for app: TestCodeInsight to get un-deployed
2021/12/04 07:12:03 Waiting for app: TestCodeInsight to get un-deployed
2021/12/04 07:12:03 App: TestCodeInsight got un-deployed
2021/12/04 07:12:03 No eventing-con process running
2021/12/04 07:12:03 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/04 07:12:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:12:15 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 07:12:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:12:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:12:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 07:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (84.39s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.06s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12648.406s

Functional tests took 3 hours, 30 minutes and 54 seconds.

N1ql tests

2021/12/04 07:12:36 Started n1ql
2021/12/04 07:12:45 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/04 07:12:45 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/04 07:12:45 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 07:12:46 Removing node(s): [] from the cluster
2021/12/04 07:12:51 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/04 07:12:56 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/04 07:12:56 App: TestRecursiveMutationN1QL got un-deployed
2021/12/04 07:12:56 No eventing-con process running
2021/12/04 07:12:56 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/04 07:12:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:13:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:13:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:13:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:13:10 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:13:10 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.61s)
=== RUN   TestFlexReset1
2021/12/04 07:13:15 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/04 07:13:15 App: TestFlexReset1 got un-deployed
2021/12/04 07:13:15 No eventing-con process running
2021/12/04 07:13:15 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/04 07:13:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:13:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:13:23 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:13:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:13:28 Starting bucket ops to source bucket
2021/12/04 07:13:29 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:13:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/04 07:13:35 Waiting for app: TestFlexReset1 to get deployed
2021/12/04 07:13:40 Waiting for app: TestFlexReset1 to get deployed
2021/12/04 07:13:45 Waiting for app: TestFlexReset1 to get deployed
2021/12/04 07:13:50 Waiting for app: TestFlexReset1 to get deployed
2021/12/04 07:13:50 App: TestFlexReset1 got deployed
2021/12/04 07:13:55 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/04 07:13:55 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/04 07:13:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:13:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:14:00 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/04 07:14:05 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/04 07:14:10 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/04 07:14:15 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/04 07:14:15 App: TestFlexReset1 got un-deployed
2021/12/04 07:14:15 No eventing-con process running
2021/12/04 07:14:15 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/04 07:14:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:14:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 07:14:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:14:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:14:39 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 07:14:44 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/04 07:14:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.53s)
=== RUN   TestFlexReset2
2021/12/04 07:14:49 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/04 07:14:49 App: TestFlexReset2 got un-deployed
2021/12/04 07:14:49 No eventing-con process running
2021/12/04 07:14:49 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/04 07:14:51 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:14:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:14:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:15:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:15:03 Starting bucket ops to source bucket
2021/12/04 07:15:04 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:15:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/04 07:15:10 Waiting for app: TestFlexReset2 to get deployed
2021/12/04 07:15:15 Waiting for app: TestFlexReset2 to get deployed
2021/12/04 07:15:20 Waiting for app: TestFlexReset2 to get deployed
2021/12/04 07:15:25 Waiting for app: TestFlexReset2 to get deployed
2021/12/04 07:15:25 App: TestFlexReset2 got deployed
2021/12/04 07:15:30 Waiting for dst bucket item count to get to: 1000 curr count: 531
2021/12/04 07:15:35 Waiting for dst bucket item count to get to: 1000 curr count: 998
2021/12/04 07:15:35 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/04 07:15:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:15:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:15:40 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/04 07:15:45 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/04 07:15:50 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/04 07:15:55 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/04 07:15:55 App: TestFlexReset2 got un-deployed
2021/12/04 07:15:55 No eventing-con process running
2021/12/04 07:15:55 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/04 07:15:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:16:07 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 07:16:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:16:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:16:19 Waiting for dst bucket item count to get to: 0 curr count: 857
2021/12/04 07:16:24 Waiting for dst bucket item count to get to: 0 curr count: 357
2021/12/04 07:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.44s)
=== RUN   TestN1QLLabelledBreak
2021/12/04 07:16:29 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/04 07:16:29 App: TestN1QLLabelledBreak got un-deployed
2021/12/04 07:16:29 No eventing-con process running
2021/12/04 07:16:29 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/04 07:16:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:16:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:16:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:16:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:16:42 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:16:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/04 07:16:48 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/04 07:16:53 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/04 07:16:58 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/04 07:17:03 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/04 07:17:03 App: TestN1QLLabelledBreak got deployed
2021/12/04 07:17:03 Starting bucket ops to source bucket
2021/12/04 07:17:13 Waiting for dst bucket item count to get to: 10000 curr count: 259
2021/12/04 07:17:18 Waiting for dst bucket item count to get to: 10000 curr count: 793
2021/12/04 07:17:23 Waiting for dst bucket item count to get to: 10000 curr count: 793
2021/12/04 07:17:28 Waiting for dst bucket item count to get to: 10000 curr count: 2662
2021/12/04 07:17:33 Waiting for dst bucket item count to get to: 10000 curr count: 4703
2021/12/04 07:17:38 Waiting for dst bucket item count to get to: 10000 curr count: 6738
2021/12/04 07:17:43 Waiting for dst bucket item count to get to: 10000 curr count: 8747
2021/12/04 07:17:48 Waiting for dst bucket item count to get to: 10000 curr count: 9584
2021/12/04 07:17:48 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 07:17:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:17:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:17:53 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/04 07:17:58 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/04 07:18:03 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/04 07:18:08 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/04 07:18:08 App: TestN1QLLabelledBreak got un-deployed
2021/12/04 07:18:08 No eventing-con process running
2021/12/04 07:18:08 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/04 07:18:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:18:20 Waiting for dst bucket item count to get to: 0 curr count: 3645
2021/12/04 07:18:25 Waiting for dst bucket item count to get to: 0 curr count: 3645
2021/12/04 07:18:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:18:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:18:37 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 07:18:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (133.48s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/04 07:18:42 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/04 07:18:42 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/04 07:18:42 No eventing-con process running
2021/12/04 07:18:42 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/04 07:18:44 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:18:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:18:51 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:18:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:18:56 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:18:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/04 07:19:02 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/04 07:19:07 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/04 07:19:12 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/04 07:19:17 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/04 07:19:17 App: TestN1QLUnlabelledBreak got deployed
2021/12/04 07:19:17 Starting bucket ops to source bucket
2021/12/04 07:19:27 Waiting for dst bucket item count to get to: 10000 curr count: 234
2021/12/04 07:19:32 Waiting for dst bucket item count to get to: 10000 curr count: 234
2021/12/04 07:19:37 Waiting for dst bucket item count to get to: 10000 curr count: 4773
2021/12/04 07:19:42 Waiting for dst bucket item count to get to: 10000 curr count: 4773
2021/12/04 07:19:47 Waiting for dst bucket item count to get to: 10000 curr count: 4773
2021/12/04 07:19:52 Waiting for dst bucket item count to get to: 10000 curr count: 9948
2021/12/04 07:19:57 Waiting for dst bucket item count to get to: 10000 curr count: 9948
2021/12/04 07:19:57 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 07:19:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:19:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:20:02 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/04 07:20:07 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/04 07:20:12 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/04 07:20:17 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/04 07:20:17 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/04 07:20:17 No eventing-con process running
2021/12/04 07:20:17 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/04 07:20:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:20:29 Waiting for dst bucket item count to get to: 0 curr count: 4890
2021/12/04 07:20:34 Waiting for dst bucket item count to get to: 0 curr count: 2390
2021/12/04 07:20:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:20:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:20:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 07:20:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (128.60s)
=== RUN   TestN1QLThrowStatement
2021/12/04 07:20:51 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/04 07:20:51 App: TestN1QLThrowStatement got un-deployed
2021/12/04 07:20:51 No eventing-con process running
2021/12/04 07:20:51 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/04 07:20:53 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:20:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:20:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:21:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:21:04 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:21:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/04 07:21:10 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/04 07:21:16 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/04 07:21:21 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/04 07:21:26 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/04 07:21:26 App: TestN1QLThrowStatement got deployed
2021/12/04 07:21:26 Starting bucket ops to source bucket
2021/12/04 07:21:35 Waiting for dst bucket item count to get to: 10000 curr count: 355
2021/12/04 07:21:40 Waiting for dst bucket item count to get to: 10000 curr count: 355
2021/12/04 07:21:45 Waiting for dst bucket item count to get to: 10000 curr count: 355
2021/12/04 07:21:50 Waiting for dst bucket item count to get to: 10000 curr count: 3884
2021/12/04 07:21:55 Waiting for dst bucket item count to get to: 10000 curr count: 3884
2021/12/04 07:22:00 Waiting for dst bucket item count to get to: 10000 curr count: 7828
2021/12/04 07:22:05 Waiting for dst bucket item count to get to: 10000 curr count: 7828
2021/12/04 07:22:05 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 07:22:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:22:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:22:10 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/04 07:22:15 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/04 07:22:20 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/04 07:22:25 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/04 07:22:25 App: TestN1QLThrowStatement got un-deployed
2021/12/04 07:22:25 No eventing-con process running
2021/12/04 07:22:25 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/04 07:22:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:22:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 07:22:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:22:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:22:49 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 07:22:54 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 07:22:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (128.29s)
=== RUN   TestN1QLNestedForLoop
2021/12/04 07:22:59 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/04 07:22:59 App: TestN1QLNestedForLoop got un-deployed
2021/12/04 07:22:59 No eventing-con process running
2021/12/04 07:22:59 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/04 07:23:01 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:23:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:23:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:23:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:23:13 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:23:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/04 07:23:19 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/04 07:23:24 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/04 07:23:29 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/04 07:23:34 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/04 07:23:34 App: TestN1QLNestedForLoop got deployed
2021/12/04 07:23:34 Starting bucket ops to source bucket
2021/12/04 07:23:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 07:23:48 Waiting for dst bucket item count to get to: 5000 curr count: 119
2021/12/04 07:23:53 Waiting for dst bucket item count to get to: 5000 curr count: 389
2021/12/04 07:23:58 Waiting for dst bucket item count to get to: 5000 curr count: 1479
2021/12/04 07:24:03 Waiting for dst bucket item count to get to: 5000 curr count: 2680
2021/12/04 07:24:08 Waiting for dst bucket item count to get to: 5000 curr count: 3901
2021/12/04 07:24:13 Waiting for dst bucket item count to get to: 5000 curr count: 4930
2021/12/04 07:24:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 07:24:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:24:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:24:18 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/04 07:24:23 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/04 07:24:28 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/04 07:24:33 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/04 07:24:34 App: TestN1QLNestedForLoop got un-deployed
2021/12/04 07:24:34 No eventing-con process running
2021/12/04 07:24:34 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/04 07:24:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:24:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 07:24:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:24:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:24:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 07:24:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (123.20s)
=== RUN   TestN1QLAutoClose
2021/12/04 07:25:02 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/04 07:25:02 App: TestN1QLAutoClose got un-deployed
2021/12/04 07:25:02 No eventing-con process running
2021/12/04 07:25:02 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/04 07:25:04 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:25:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:25:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:25:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:25:16 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:25:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/04 07:25:22 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/04 07:25:27 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/04 07:25:32 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/04 07:25:37 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/04 07:25:37 App: TestN1QLAutoClose got deployed
2021/12/04 07:25:37 Starting bucket ops to source bucket
2021/12/04 07:25:43 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/04 07:25:48 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/04 07:25:53 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/04 07:25:58 Waiting for dst bucket item count to get to: 500 curr count: 23
2021/12/04 07:26:03 Waiting for dst bucket item count to get to: 500 curr count: 23
2021/12/04 07:26:08 Waiting for dst bucket item count to get to: 500 curr count: 45
2021/12/04 07:26:13 Waiting for dst bucket item count to get to: 500 curr count: 45
2021/12/04 07:26:18 Waiting for dst bucket item count to get to: 500 curr count: 67
2021/12/04 07:26:23 Waiting for dst bucket item count to get to: 500 curr count: 67
2021/12/04 07:26:28 Waiting for dst bucket item count to get to: 500 curr count: 88
2021/12/04 07:26:33 Waiting for dst bucket item count to get to: 500 curr count: 88
2021/12/04 07:26:38 Waiting for dst bucket item count to get to: 500 curr count: 109
2021/12/04 07:26:43 Waiting for dst bucket item count to get to: 500 curr count: 109
2021/12/04 07:26:48 Waiting for dst bucket item count to get to: 500 curr count: 119
2021/12/04 07:26:53 Waiting for dst bucket item count to get to: 500 curr count: 129
2021/12/04 07:26:58 Waiting for dst bucket item count to get to: 500 curr count: 140
2021/12/04 07:27:03 Waiting for dst bucket item count to get to: 500 curr count: 151
2021/12/04 07:27:08 Waiting for dst bucket item count to get to: 500 curr count: 161
2021/12/04 07:27:13 Waiting for dst bucket item count to get to: 500 curr count: 173
2021/12/04 07:27:18 Waiting for dst bucket item count to get to: 500 curr count: 184
2021/12/04 07:27:23 Waiting for dst bucket item count to get to: 500 curr count: 196
2021/12/04 07:27:28 Waiting for dst bucket item count to get to: 500 curr count: 206
2021/12/04 07:27:33 Waiting for dst bucket item count to get to: 500 curr count: 217
2021/12/04 07:27:38 Waiting for dst bucket item count to get to: 500 curr count: 227
2021/12/04 07:27:43 Waiting for dst bucket item count to get to: 500 curr count: 238
2021/12/04 07:27:48 Waiting for dst bucket item count to get to: 500 curr count: 247
2021/12/04 07:27:53 Waiting for dst bucket item count to get to: 500 curr count: 259
2021/12/04 07:27:58 Waiting for dst bucket item count to get to: 500 curr count: 269
2021/12/04 07:28:03 Waiting for dst bucket item count to get to: 500 curr count: 281
2021/12/04 07:28:08 Waiting for dst bucket item count to get to: 500 curr count: 293
2021/12/04 07:28:13 Waiting for dst bucket item count to get to: 500 curr count: 302
2021/12/04 07:28:18 Waiting for dst bucket item count to get to: 500 curr count: 323
2021/12/04 07:28:23 Waiting for dst bucket item count to get to: 500 curr count: 323
2021/12/04 07:28:28 Waiting for dst bucket item count to get to: 500 curr count: 344
2021/12/04 07:28:33 Waiting for dst bucket item count to get to: 500 curr count: 344
2021/12/04 07:28:38 Waiting for dst bucket item count to get to: 500 curr count: 364
2021/12/04 07:28:43 Waiting for dst bucket item count to get to: 500 curr count: 364
2021/12/04 07:28:48 Waiting for dst bucket item count to get to: 500 curr count: 383
2021/12/04 07:28:53 Waiting for dst bucket item count to get to: 500 curr count: 383
2021/12/04 07:28:58 Waiting for dst bucket item count to get to: 500 curr count: 406
2021/12/04 07:29:03 Waiting for dst bucket item count to get to: 500 curr count: 406
2021/12/04 07:29:08 Waiting for dst bucket item count to get to: 500 curr count: 428
2021/12/04 07:29:13 Waiting for dst bucket item count to get to: 500 curr count: 428
2021/12/04 07:29:18 Waiting for dst bucket item count to get to: 500 curr count: 449
2021/12/04 07:29:23 Waiting for dst bucket item count to get to: 500 curr count: 449
2021/12/04 07:29:28 Waiting for dst bucket item count to get to: 500 curr count: 471
2021/12/04 07:29:33 Waiting for dst bucket item count to get to: 500 curr count: 471
2021/12/04 07:29:38 Waiting for dst bucket item count to get to: 500 curr count: 491
2021/12/04 07:29:43 Waiting for dst bucket item count to get to: 500 curr count: 491
2021/12/04 07:29:43 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/04 07:29:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:29:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:29:48 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/04 07:29:53 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/04 07:29:58 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/04 07:30:03 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/04 07:30:03 App: TestN1QLAutoClose got un-deployed
2021/12/04 07:30:03 No eventing-con process running
2021/12/04 07:30:03 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/04 07:30:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:30:15 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/04 07:30:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:30:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:30:27 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/04 07:30:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.67s)
=== RUN   TestN1QLPosParams
2021/12/04 07:30:32 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/04 07:30:32 App: TestN1QLPosParams got un-deployed
2021/12/04 07:30:32 No eventing-con process running
2021/12/04 07:30:32 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/04 07:30:34 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:30:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:30:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:30:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:30:46 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:30:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/04 07:30:52 Waiting for app: TestN1QLPosParams to get deployed
2021/12/04 07:30:57 Waiting for app: TestN1QLPosParams to get deployed
2021/12/04 07:31:02 Waiting for app: TestN1QLPosParams to get deployed
2021/12/04 07:31:07 Waiting for app: TestN1QLPosParams to get deployed
2021/12/04 07:31:07 App: TestN1QLPosParams got deployed
2021/12/04 07:31:07 Starting bucket ops to source bucket
2021/12/04 07:31:16 Waiting for dst bucket item count to get to: 5000 curr count: 464
2021/12/04 07:31:21 Waiting for dst bucket item count to get to: 5000 curr count: 464
2021/12/04 07:31:21 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 07:31:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:31:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:31:27 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/04 07:31:32 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/04 07:31:37 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/04 07:31:42 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/04 07:31:42 App: TestN1QLPosParams got un-deployed
2021/12/04 07:31:42 No eventing-con process running
2021/12/04 07:31:42 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/04 07:31:43 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:31:53 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 07:31:58 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 07:31:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:32:00 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:32:10 Waiting for dst bucket item count to get to: 0 curr count: 3623
2021/12/04 07:32:15 Waiting for dst bucket item count to get to: 0 curr count: 3623
2021/12/04 07:32:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.42s)
=== RUN   TestN1QLExhaustConnPool
2021/12/04 07:32:20 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/04 07:32:20 App: TestN1QLExhaustConnPool got un-deployed
2021/12/04 07:32:20 No eventing-con process running
2021/12/04 07:32:20 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/04 07:32:22 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:32:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:32:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:32:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:32:34 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:32:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/04 07:32:40 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/04 07:32:45 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/04 07:32:50 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/04 07:32:55 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/04 07:32:55 App: TestN1QLExhaustConnPool got deployed
2021/12/04 07:32:55 Starting bucket ops to source bucket
2021/12/04 07:33:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 07:33:05 Waiting for dst bucket item count to get to: 100 curr count: 29
2021/12/04 07:33:10 Waiting for dst bucket item count to get to: 100 curr count: 79
2021/12/04 07:33:10 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 07:33:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:33:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:33:16 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/04 07:33:21 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/04 07:33:26 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/04 07:33:31 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/04 07:33:31 App: TestN1QLExhaustConnPool got un-deployed
2021/12/04 07:33:31 No eventing-con process running
2021/12/04 07:33:31 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/04 07:33:32 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:33:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 07:33:47 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 07:33:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:33:49 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:33:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 07:34:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 07:34:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.94s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/04 07:34:09 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/04 07:34:09 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/04 07:34:09 No eventing-con process running
2021/12/04 07:34:09 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/04 07:34:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:34:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:34:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:34:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:34:23 Starting bucket ops to source bucket
2021/12/04 07:34:41 Starting bucket ops to source bucket
2021/12/04 07:34:58 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 12864
2021/12/04 07:35:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/04 07:35:05 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/04 07:35:10 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/04 07:35:15 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/04 07:35:20 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/04 07:35:20 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/04 07:35:20 Starting bucket ops to source bucket
2021/12/04 07:35:25 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 07:35:30 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 07:35:35 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 07:35:40 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 07:35:45 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 07:35:50 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 07:35:55 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 07:36:00 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 07:36:05 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 07:36:10 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 07:36:15 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 07:36:15 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/04 07:36:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:36:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:36:20 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/04 07:36:25 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/04 07:36:30 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/04 07:36:35 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/04 07:36:35 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/04 07:36:35 No eventing-con process running
2021/12/04 07:36:35 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/04 07:36:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:36:47 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/04 07:36:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:36:49 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:36:59 Waiting for dst bucket item count to get to: 0 curr count: 28122
2021/12/04 07:37:04 Waiting for dst bucket item count to get to: 0 curr count: 13122
2021/12/04 07:37:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.57s)
=== RUN   TestN1QLQueryConsistency
2021/12/04 07:37:09 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/04 07:37:09 App: n1qlquery_consistency got un-deployed
2021/12/04 07:37:09 No eventing-con process running
2021/12/04 07:37:09 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/04 07:37:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:37:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:37:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:37:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:37:22 Starting bucket ops to source bucket
2021/12/04 07:37:26 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:37:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/04 07:37:32 Waiting for app: n1qlquery_consistency to get deployed
2021/12/04 07:37:37 Waiting for app: n1qlquery_consistency to get deployed
2021/12/04 07:37:42 Waiting for app: n1qlquery_consistency to get deployed
2021/12/04 07:37:47 Waiting for app: n1qlquery_consistency to get deployed
2021/12/04 07:37:47 App: n1qlquery_consistency got deployed
2021/12/04 07:37:52 Waiting for dst bucket item count to get to: 5000 curr count: 105
2021/12/04 07:37:57 Waiting for dst bucket item count to get to: 5000 curr count: 2605
2021/12/04 07:37:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 07:37:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 07:37:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:38:02 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/04 07:38:07 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/04 07:38:12 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/04 07:38:17 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/04 07:38:17 App: n1qlquery_consistency got un-deployed
2021/12/04 07:38:17 No eventing-con process running
2021/12/04 07:38:17 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/04 07:38:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:38:29 Waiting for dst bucket item count to get to: 0 curr count: 4848
2021/12/04 07:38:34 Waiting for dst bucket item count to get to: 0 curr count: 2348
2021/12/04 07:38:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:38:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 07:38:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 07:38:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.89s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1566.072s

N1ql tests took 0 hours, 26 minutes and 10 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/04 07:40:21 Started rebalance
=== RUN   TestDuplicateWithEventingRebIn
2021/12/04 07:40:38 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/04 07:40:38 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/04 07:40:38 No eventing-con process running
2021/12/04 07:40:38 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/04 07:40:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:40:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:40:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:40:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:40:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:40:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:40:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:40:57 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:40:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 07:41:03 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/04 07:41:08 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/04 07:41:13 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/04 07:41:18 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/04 07:41:18 App: TestDuplicateWithEventingRebIn got deployed
2021/12/04 07:41:18 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 07:41:18 Starting bucket ops to source bucket
2021/12/04 07:41:33 Removing node(s): [] from the cluster
2021/12/04 07:41:38 Rebalance progress: 25
2021/12/04 07:41:43 Rebalance progress: 25
2021/12/04 07:41:48 Rebalance progress: 25
2021/12/04 07:41:53 Rebalance progress: 25
2021/12/04 07:41:58 Rebalance progress: 75
2021/12/04 07:42:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:42:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:42:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 07:42:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/04 07:42:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 07:42:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 07:42:08 Rebalance progress: 25
2021/12/04 07:42:13 Rebalance progress: 25
2021/12/04 07:42:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:42:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:42:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 07:42:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/04 07:42:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 07:42:18 Waiting for src bucket item count to get to: 200000 curr count: 79624
2021/12/04 07:42:23 Waiting for src bucket item count to get to: 200000 curr count: 79624
2021/12/04 07:42:28 Waiting for src bucket item count to get to: 200000 curr count: 95698
2021/12/04 07:42:33 Waiting for src bucket item count to get to: 200000 curr count: 95698
2021/12/04 07:42:38 Waiting for src bucket item count to get to: 200000 curr count: 111894
2021/12/04 07:42:43 Waiting for src bucket item count to get to: 200000 curr count: 111894
2021/12/04 07:42:48 Waiting for src bucket item count to get to: 200000 curr count: 128359
2021/12/04 07:42:53 Waiting for src bucket item count to get to: 200000 curr count: 128359
2021/12/04 07:42:58 Waiting for src bucket item count to get to: 200000 curr count: 144824
2021/12/04 07:43:03 Waiting for src bucket item count to get to: 200000 curr count: 144824
2021/12/04 07:43:08 Waiting for src bucket item count to get to: 200000 curr count: 161125
2021/12/04 07:43:13 Waiting for src bucket item count to get to: 200000 curr count: 161125
2021/12/04 07:43:18 Waiting for src bucket item count to get to: 200000 curr count: 177359
2021/12/04 07:43:23 Waiting for src bucket item count to get to: 200000 curr count: 177359
2021/12/04 07:43:28 Waiting for src bucket item count to get to: 200000 curr count: 193787
2021/12/04 07:43:33 Waiting for src bucket item count to get to: 200000 curr count: 193787
2021/12/04 07:43:43 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 07:43:48 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 07:43:53 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 07:43:58 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 07:44:03 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 07:44:08 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 07:44:13 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 07:44:18 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 07:44:23 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 07:44:28 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 07:44:28 Post SET ops. Expected item count: 200000 got 200001
2021/12/04 07:44:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:44:33 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/04 07:44:38 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/04 07:44:43 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/04 07:44:48 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/04 07:44:48 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/04 07:44:48 No eventing-con process running
2021/12/04 07:44:48 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/04 07:44:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:44:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:45:00 Waiting for dst bucket item count to get to: 0 curr count: 9938
2021/12/04 07:45:05 Waiting for dst bucket item count to get to: 0 curr count: 9938
2021/12/04 07:45:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:45:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:45:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:45:17 Waiting for dst bucket item count to get to: 0 curr count: 200001
2021/12/04 07:45:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (288.47s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/04 07:45:22 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 07:45:22 Removing node(s): [] from the cluster
2021/12/04 07:45:27 Rebalance progress: 100
2021/12/04 07:45:32 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/04 07:45:32 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/04 07:45:32 No eventing-con process running
2021/12/04 07:45:32 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/04 07:45:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:45:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:45:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:45:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:45:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:45:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:45:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:45:51 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:45:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 07:45:57 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/04 07:46:02 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/04 07:46:07 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/04 07:46:12 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/04 07:46:12 App: TestDuplicateWithEventingRebOut got deployed
2021/12/04 07:46:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 07:46:12 Starting bucket ops to source bucket
2021/12/04 07:46:17 Rebalance progress: 25
2021/12/04 07:46:22 Rebalance progress: 25
2021/12/04 07:46:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:46:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:46:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 07:46:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/04 07:46:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 07:46:27 Waiting for src bucket item count to get to: 200000 curr count: 11628
2021/12/04 07:46:32 Waiting for src bucket item count to get to: 200000 curr count: 27838
2021/12/04 07:46:37 Waiting for src bucket item count to get to: 200000 curr count: 27838
2021/12/04 07:46:42 Waiting for src bucket item count to get to: 200000 curr count: 43944
2021/12/04 07:46:47 Waiting for src bucket item count to get to: 200000 curr count: 43944
2021/12/04 07:46:52 Waiting for src bucket item count to get to: 200000 curr count: 60248
2021/12/04 07:46:57 Waiting for src bucket item count to get to: 200000 curr count: 60248
2021/12/04 07:47:02 Waiting for src bucket item count to get to: 200000 curr count: 60248
2021/12/04 07:47:07 Waiting for src bucket item count to get to: 200000 curr count: 76473
2021/12/04 07:47:12 Waiting for src bucket item count to get to: 200000 curr count: 76473
2021/12/04 07:47:17 Waiting for src bucket item count to get to: 200000 curr count: 92403
2021/12/04 07:47:22 Waiting for src bucket item count to get to: 200000 curr count: 92403
2021/12/04 07:47:27 Waiting for src bucket item count to get to: 200000 curr count: 108741
2021/12/04 07:47:32 Waiting for src bucket item count to get to: 200000 curr count: 108741
2021/12/04 07:47:37 Waiting for src bucket item count to get to: 200000 curr count: 125112
2021/12/04 07:47:42 Waiting for src bucket item count to get to: 200000 curr count: 125112
2021/12/04 07:47:47 Waiting for src bucket item count to get to: 200000 curr count: 141644
2021/12/04 07:47:52 Waiting for src bucket item count to get to: 200000 curr count: 141644
2021/12/04 07:47:57 Waiting for src bucket item count to get to: 200000 curr count: 158355
2021/12/04 07:48:02 Waiting for src bucket item count to get to: 200000 curr count: 158355
2021/12/04 07:48:07 Waiting for src bucket item count to get to: 200000 curr count: 174523
2021/12/04 07:48:12 Waiting for src bucket item count to get to: 200000 curr count: 174523
2021/12/04 07:48:17 Waiting for src bucket item count to get to: 200000 curr count: 190784
2021/12/04 07:48:22 Waiting for src bucket item count to get to: 200000 curr count: 190784
2021/12/04 07:48:27 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 07:48:27 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 07:48:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:48:32 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/04 07:48:37 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/04 07:48:42 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/04 07:48:47 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/04 07:48:47 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/04 07:48:47 No eventing-con process running
2021/12/04 07:48:47 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/04 07:48:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:48:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:48:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 07:48:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:49:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:49:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:49:11 Waiting for dst bucket item count to get to: 0 curr count: 121181
2021/12/04 07:49:16 Waiting for dst bucket item count to get to: 0 curr count: 121181
2021/12/04 07:49:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (238.77s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/04 07:49:26 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 07:49:26 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/04 07:49:26 No eventing-con process running
2021/12/04 07:49:26 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/04 07:49:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:49:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:49:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:49:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:49:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:49:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:49:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:49:44 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:49:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 07:49:50 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/04 07:49:55 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/04 07:50:00 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/04 07:50:05 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/04 07:50:05 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/04 07:50:05 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 07:50:05 Starting bucket ops to source bucket
2021/12/04 07:50:06 Removing node(s): [] from the cluster
2021/12/04 07:50:11 Rebalance progress: 25
2021/12/04 07:50:16 Rebalance progress: 25
2021/12/04 07:50:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:50:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:50:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 07:50:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/04 07:50:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 07:50:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 07:50:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 07:50:33 Rebalance progress: 16.66666666666667
2021/12/04 07:50:38 Rebalance progress: 16.66666666666667
2021/12/04 07:50:43 Rebalance progress: 16.66666666666667
2021/12/04 07:50:48 Rebalance progress: 72.22222222222223
2021/12/04 07:50:53 Rebalance progress: 72.22222222222223
2021/12/04 07:50:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:50:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:50:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 07:50:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 07:50:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 07:51:03 Rebalance progress: 16.66666666666667
2021/12/04 07:51:08 Rebalance progress: 16.66666666666667
2021/12/04 07:51:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:51:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:51:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 07:51:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 07:51:13 Waiting for src bucket item count to get to: 200000 curr count: 97387
2021/12/04 07:51:18 Waiting for src bucket item count to get to: 200000 curr count: 97387
2021/12/04 07:51:23 Waiting for src bucket item count to get to: 200000 curr count: 113437
2021/12/04 07:51:28 Waiting for src bucket item count to get to: 200000 curr count: 113437
2021/12/04 07:51:33 Waiting for src bucket item count to get to: 200000 curr count: 129695
2021/12/04 07:51:38 Waiting for src bucket item count to get to: 200000 curr count: 129695
2021/12/04 07:51:43 Waiting for src bucket item count to get to: 200000 curr count: 145767
2021/12/04 07:51:48 Waiting for src bucket item count to get to: 200000 curr count: 145767
2021/12/04 07:51:53 Waiting for src bucket item count to get to: 200000 curr count: 162027
2021/12/04 07:51:58 Waiting for src bucket item count to get to: 200000 curr count: 162027
2021/12/04 07:52:03 Waiting for src bucket item count to get to: 200000 curr count: 177975
2021/12/04 07:52:08 Waiting for src bucket item count to get to: 200000 curr count: 177975
2021/12/04 07:52:13 Waiting for src bucket item count to get to: 200000 curr count: 194084
2021/12/04 07:52:18 Waiting for src bucket item count to get to: 200000 curr count: 194084
2021/12/04 07:52:23 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 07:52:23 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 07:52:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:52:28 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 07:52:33 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 07:52:38 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 07:52:43 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 07:52:43 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/04 07:52:43 No eventing-con process running
2021/12/04 07:52:43 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/04 07:52:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:52:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:52:55 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 07:52:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:52:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:53:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:53:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 07:53:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (231.18s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/04 07:53:17 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 07:53:17 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/04 07:53:17 No eventing-con process running
2021/12/04 07:53:17 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/04 07:53:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:53:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:53:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:53:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:53:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:53:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:53:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:53:35 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:53:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 07:53:41 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 07:53:46 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 07:53:51 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 07:53:56 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 07:53:56 App: TestDuplicateWithKVRebIn got deployed
2021/12/04 07:53:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 07:53:56 Starting bucket ops to source bucket
2021/12/04 07:53:57 Removing node(s): [] from the cluster
2021/12/04 07:54:02 Rebalance progress: 16.66666666666667
2021/12/04 07:54:07 Rebalance progress: 16.66666666666667
2021/12/04 07:54:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:54:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:54:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 07:54:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 07:54:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 07:54:17 Rebalance progress: 16.66666666666667
2021/12/04 07:54:22 Rebalance progress: 16.66666666666667
2021/12/04 07:54:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 07:54:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 07:54:27 Waiting for src bucket item count to get to: 200000 curr count: 34933
2021/12/04 07:54:32 Waiting for src bucket item count to get to: 200000 curr count: 50523
2021/12/04 07:54:37 Waiting for src bucket item count to get to: 200000 curr count: 50523
2021/12/04 07:54:42 Waiting for src bucket item count to get to: 200000 curr count: 66765
2021/12/04 07:54:47 Waiting for src bucket item count to get to: 200000 curr count: 66765
2021/12/04 07:54:52 Waiting for src bucket item count to get to: 200000 curr count: 82194
2021/12/04 07:54:57 Waiting for src bucket item count to get to: 200000 curr count: 82194
2021/12/04 07:55:02 Waiting for src bucket item count to get to: 200000 curr count: 97643
2021/12/04 07:55:07 Waiting for src bucket item count to get to: 200000 curr count: 97643
2021/12/04 07:55:12 Waiting for src bucket item count to get to: 200000 curr count: 97643
2021/12/04 07:55:17 Waiting for src bucket item count to get to: 200000 curr count: 113377
2021/12/04 07:55:22 Waiting for src bucket item count to get to: 200000 curr count: 113377
2021/12/04 07:55:27 Waiting for src bucket item count to get to: 200000 curr count: 129483
2021/12/04 07:55:32 Waiting for src bucket item count to get to: 200000 curr count: 129483
2021/12/04 07:55:37 Waiting for src bucket item count to get to: 200000 curr count: 145716
2021/12/04 07:55:42 Waiting for src bucket item count to get to: 200000 curr count: 145716
2021/12/04 07:55:47 Waiting for src bucket item count to get to: 200000 curr count: 161678
2021/12/04 07:55:52 Waiting for src bucket item count to get to: 200000 curr count: 161678
2021/12/04 07:55:57 Waiting for src bucket item count to get to: 200000 curr count: 176895
2021/12/04 07:56:02 Waiting for src bucket item count to get to: 200000 curr count: 176895
2021/12/04 07:56:07 Waiting for src bucket item count to get to: 200000 curr count: 192598
2021/12/04 07:56:12 Waiting for src bucket item count to get to: 200000 curr count: 192598
2021/12/04 07:56:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 07:56:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 07:56:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 07:56:22 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 07:56:27 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 07:56:32 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 07:56:37 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 07:56:37 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/04 07:56:37 No eventing-con process running
2021/12/04 07:56:37 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/04 07:56:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:56:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:56:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 07:56:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:56:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:56:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:57:01 Waiting for dst bucket item count to get to: 0 curr count: 164606
2021/12/04 07:57:06 Waiting for dst bucket item count to get to: 0 curr count: 164606
2021/12/04 07:57:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (239.14s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/04 07:57:11 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 07:57:11 Removing node(s): [] from the cluster
2021/12/04 07:57:16 Rebalance progress: 100
2021/12/04 07:57:21 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 07:57:21 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/04 07:57:21 No eventing-con process running
2021/12/04 07:57:21 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/04 07:57:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:57:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:57:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 07:57:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:57:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:57:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:57:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 07:57:40 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 07:57:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 07:57:46 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 07:57:51 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 07:57:56 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 07:58:01 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 07:58:01 App: TestDuplicateWithKVRebOut got deployed
2021/12/04 07:58:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 07:58:01 Starting bucket ops to source bucket
2021/12/04 07:58:06 Rebalance progress: 16.66666666666667
2021/12/04 07:58:11 Rebalance progress: 16.66666666666667
2021/12/04 07:58:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:58:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 07:58:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 07:58:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 07:58:16 Waiting for src bucket item count to get to: 200000 curr count: 12601
2021/12/04 07:58:21 Waiting for src bucket item count to get to: 200000 curr count: 28735
2021/12/04 07:58:26 Waiting for src bucket item count to get to: 200000 curr count: 28735
2021/12/04 07:58:31 Waiting for src bucket item count to get to: 200000 curr count: 45008
2021/12/04 07:58:36 Waiting for src bucket item count to get to: 200000 curr count: 45008
2021/12/04 07:58:41 Waiting for src bucket item count to get to: 200000 curr count: 61179
2021/12/04 07:58:46 Waiting for src bucket item count to get to: 200000 curr count: 61179
2021/12/04 07:58:51 Waiting for src bucket item count to get to: 200000 curr count: 61179
2021/12/04 07:58:56 Waiting for src bucket item count to get to: 200000 curr count: 76849
2021/12/04 07:59:01 Waiting for src bucket item count to get to: 200000 curr count: 76849
2021/12/04 07:59:06 Waiting for src bucket item count to get to: 200000 curr count: 92909
2021/12/04 07:59:11 Waiting for src bucket item count to get to: 200000 curr count: 92909
2021/12/04 07:59:16 Waiting for src bucket item count to get to: 200000 curr count: 108866
2021/12/04 07:59:21 Waiting for src bucket item count to get to: 200000 curr count: 108866
2021/12/04 07:59:26 Waiting for src bucket item count to get to: 200000 curr count: 124741
2021/12/04 07:59:31 Waiting for src bucket item count to get to: 200000 curr count: 124741
2021/12/04 07:59:36 Waiting for src bucket item count to get to: 200000 curr count: 140879
2021/12/04 07:59:41 Waiting for src bucket item count to get to: 200000 curr count: 140879
2021/12/04 07:59:46 Waiting for src bucket item count to get to: 200000 curr count: 157207
2021/12/04 07:59:51 Waiting for src bucket item count to get to: 200000 curr count: 157207
2021/12/04 07:59:56 Waiting for src bucket item count to get to: 200000 curr count: 172681
2021/12/04 08:00:01 Waiting for src bucket item count to get to: 200000 curr count: 172681
2021/12/04 08:00:06 Waiting for src bucket item count to get to: 200000 curr count: 188621
2021/12/04 08:00:11 Waiting for src bucket item count to get to: 200000 curr count: 188621
2021/12/04 08:00:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 08:00:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 08:00:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:00:21 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 08:00:26 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 08:00:31 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 08:00:36 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 08:00:36 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/04 08:00:36 No eventing-con process running
2021/12/04 08:00:36 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/04 08:00:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:00:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:00:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 08:00:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:00:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:00:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:01:00 Waiting for dst bucket item count to get to: 0 curr count: 799
2021/12/04 08:01:05 Waiting for dst bucket item count to get to: 0 curr count: 799
2021/12/04 08:01:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (239.05s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/04 08:01:15 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 08:01:15 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/04 08:01:15 No eventing-con process running
2021/12/04 08:01:15 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/04 08:01:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:01:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:01:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:01:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:01:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:01:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:01:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:01:34 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:01:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 08:01:40 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 08:01:45 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 08:01:50 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 08:01:55 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 08:01:55 App: TestDuplicateWithKVSwapReb got deployed
2021/12/04 08:01:55 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 08:01:55 Starting bucket ops to source bucket
2021/12/04 08:01:55 Removing node(s): [] from the cluster
2021/12/04 08:02:00 Rebalance progress: 16.66666666666667
2021/12/04 08:02:05 Rebalance progress: 16.66666666666667
2021/12/04 08:02:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:02:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:02:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:02:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 08:02:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 08:02:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:02:15 Rebalance progress: 16.66666666666667
2021/12/04 08:02:20 Rebalance progress: 16.66666666666667
2021/12/04 08:02:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:02:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:02:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:02:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 08:02:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 08:02:30 Rebalance progress: 16.66666666666667
2021/12/04 08:02:35 Rebalance progress: 16.66666666666667
2021/12/04 08:02:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:02:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:02:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:02:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 08:02:40 Waiting for src bucket item count to get to: 200000 curr count: 69499
2021/12/04 08:02:45 Waiting for src bucket item count to get to: 200000 curr count: 69499
2021/12/04 08:02:50 Waiting for src bucket item count to get to: 200000 curr count: 84581
2021/12/04 08:02:56 Waiting for src bucket item count to get to: 200000 curr count: 84581
2021/12/04 08:03:01 Waiting for src bucket item count to get to: 200000 curr count: 100507
2021/12/04 08:03:06 Waiting for src bucket item count to get to: 200000 curr count: 100507
2021/12/04 08:03:11 Waiting for src bucket item count to get to: 200000 curr count: 116476
2021/12/04 08:03:16 Waiting for src bucket item count to get to: 200000 curr count: 116476
2021/12/04 08:03:21 Waiting for src bucket item count to get to: 200000 curr count: 116476
2021/12/04 08:03:26 Waiting for src bucket item count to get to: 200000 curr count: 132655
2021/12/04 08:03:31 Waiting for src bucket item count to get to: 200000 curr count: 132655
2021/12/04 08:03:36 Waiting for src bucket item count to get to: 200000 curr count: 148876
2021/12/04 08:03:41 Waiting for src bucket item count to get to: 200000 curr count: 148876
2021/12/04 08:03:46 Waiting for src bucket item count to get to: 200000 curr count: 164932
2021/12/04 08:03:51 Waiting for src bucket item count to get to: 200000 curr count: 164932
2021/12/04 08:03:56 Waiting for src bucket item count to get to: 200000 curr count: 180397
2021/12/04 08:04:01 Waiting for src bucket item count to get to: 200000 curr count: 180397
2021/12/04 08:04:06 Waiting for src bucket item count to get to: 200000 curr count: 196457
2021/12/04 08:04:11 Waiting for src bucket item count to get to: 200000 curr count: 196457
2021/12/04 08:04:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 08:04:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 08:04:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:04:21 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 08:04:26 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 08:04:31 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 08:04:36 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 08:04:36 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/04 08:04:36 No eventing-con process running
2021/12/04 08:04:36 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/04 08:04:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:04:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:04:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 08:04:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:04:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:04:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:04:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.45s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/04 08:05:04 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:05:04 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 08:05:04 No eventing-con process running
2021/12/04 08:05:04 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/04 08:05:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:05:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:05:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:05:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:05:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:05:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:05:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:05:23 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:05:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/04 08:05:29 Starting bucket ops to source bucket
2021/12/04 08:05:34 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 08:05:39 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 08:05:44 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 08:05:44 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:05:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:05:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:05:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 08:05:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:05:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:05:49 Rebalance progress: 16.66666666666667
2021/12/04 08:05:54 Rebalance progress: 16.66666666666667
2021/12/04 08:05:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:05:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:05:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:05:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 08:05:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:06:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:06:05 Rebalance progress: 16.66666666666667
2021/12/04 08:06:10 Rebalance progress: 16.66666666666667
2021/12/04 08:06:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:06:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:06:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:06:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 08:06:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 08:06:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:06:25 Rebalance progress: 12.5
2021/12/04 08:06:30 Rebalance progress: 12.5
2021/12/04 08:06:35 Rebalance progress: 12.5
2021/12/04 08:06:40 Rebalance progress: 70.83333333333334
2021/12/04 08:06:45 Rebalance progress: 70.83333333333334
2021/12/04 08:06:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:06:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:06:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:06:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:06:55 Rebalance progress: 12.5
2021/12/04 08:07:00 Rebalance progress: 12.5
2021/12/04 08:07:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:07:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:07:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:07:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 08:07:10 Rebalance progress: 12.5
2021/12/04 08:07:15 Rebalance progress: 12.5
2021/12/04 08:07:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:07:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:07:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:07:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 08:07:25 Rebalance progress: 12.5
2021/12/04 08:07:30 Rebalance progress: 12.5
2021/12/04 08:07:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:07:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:07:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:07:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:07:40 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:07:45 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:07:51 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:07:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:07:56 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 08:07:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:07:56 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/04 08:07:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:08:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:08:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12626
2021/12/04 08:08:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:08:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:08:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:08:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12626
2021/12/04 08:08:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (204.78s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/04 08:08:29 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 08:08:29 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/04 08:08:29 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:08:29 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/04 08:08:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:08:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:08:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:08:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:08:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:08:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:08:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:08:48 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:08:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/04 08:08:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 08:08:59 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/04 08:09:04 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/04 08:09:09 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/04 08:09:09 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:09:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:09:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:09:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:09:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:09:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 08:09:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:09:15 Rebalance progress: 12.5
2021/12/04 08:09:20 Rebalance progress: 12.5
2021/12/04 08:09:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:09:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:09:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:09:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/04 08:09:30 Rebalance progress: 12.5
2021/12/04 08:09:35 Rebalance progress: 12.5
2021/12/04 08:09:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:09:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:09:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:09:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:09:45 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 08:09:51 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 08:09:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 08:10:01 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 08:10:01 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/04 08:10:01 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:10:01 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/04 08:10:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:10:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:10:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4630
2021/12/04 08:10:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4630
2021/12/04 08:10:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:10:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:10:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:10:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4630
2021/12/04 08:10:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (130.02s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/04 08:10:39 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 08:10:39 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/04 08:10:39 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:10:39 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/04 08:10:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:10:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:10:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:10:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:10:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:10:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:10:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:10:58 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:10:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 08:11:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 08:11:09 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/04 08:11:14 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/04 08:11:19 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/04 08:11:19 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:11:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:11:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:11:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:11:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:11:25 Rebalance progress: 12.5
2021/12/04 08:11:30 Rebalance progress: 12.5
2021/12/04 08:11:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:11:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:11:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:11:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:11:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:11:40 Rebalance progress: 12.5
2021/12/04 08:11:45 Rebalance progress: 12.5
2021/12/04 08:11:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:11:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:11:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:11:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 08:11:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:11:55 Rebalance progress: 12.5
2021/12/04 08:12:00 Rebalance progress: 12.5
2021/12/04 08:12:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:12:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:12:10 Rebalance progress: 12.5
2021/12/04 08:12:15 Rebalance progress: 12.5
2021/12/04 08:12:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:12:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:12:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:12:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 08:12:26 Rebalance progress: 12.5
2021/12/04 08:12:31 Rebalance progress: 12.5
2021/12/04 08:12:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:12:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:12:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:12:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 08:12:41 Rebalance progress: 12.5
2021/12/04 08:12:46 Rebalance progress: 12.5
2021/12/04 08:12:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:12:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:12:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:12:51 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:12:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 08:13:01 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 08:13:06 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 08:13:11 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 08:13:11 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/04 08:13:11 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:13:11 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/04 08:13:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:13:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:13:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10671
2021/12/04 08:13:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10671
2021/12/04 08:13:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:13:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:13:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:13:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (185.35s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/04 08:13:45 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 08:13:45 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/04 08:13:45 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:13:45 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/04 08:13:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:13:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:13:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:13:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:13:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:13:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:13:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:13:58 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:13:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 08:14:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 08:14:09 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/04 08:14:14 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/04 08:14:19 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/04 08:14:19 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:14:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:14:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:14:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:14:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:14:26 Rebalance progress: 12.5
2021/12/04 08:14:31 Rebalance progress: 12.5
2021/12/04 08:14:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:14:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:14:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:14:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:14:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:14:41 Rebalance progress: 12.5
2021/12/04 08:14:46 Rebalance progress: 12.5
2021/12/04 08:14:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:14:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:14:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:14:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 08:14:56 Rebalance progress: 12.5
2021/12/04 08:15:01 Rebalance progress: 12.5
2021/12/04 08:15:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:15:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:15:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:15:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:15:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 08:15:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 08:15:21 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 08:15:26 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 08:15:26 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/04 08:15:26 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:15:27 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/04 08:15:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:15:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:15:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6198
2021/12/04 08:15:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:15:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:15:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:15:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1149
2021/12/04 08:15:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1149
2021/12/04 08:15:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (140.60s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/04 08:16:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:16:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:16:05 Rebalance progress: 100
2021/12/04 08:16:10 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 08:16:10 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/04 08:16:10 No eventing-con process running
2021/12/04 08:16:10 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/04 08:16:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:16:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:16:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:16:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:16:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:16:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:16:24 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:16:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/04 08:16:30 Starting bucket ops to source bucket
2021/12/04 08:16:35 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/04 08:16:40 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/04 08:16:45 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/04 08:16:45 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:16:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:16:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:16:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:16:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:16:51 Rebalance progress: 12.5
2021/12/04 08:16:56 Rebalance progress: 12.5
2021/12/04 08:17:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:17:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:17:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:17:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/04 08:17:01 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/04 08:17:06 Rebalance progress: 12.5
2021/12/04 08:17:11 Rebalance progress: 12.5
2021/12/04 08:17:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:17:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:17:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:17:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:17:21 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 08:17:26 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 08:17:31 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 08:17:36 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 08:17:36 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/04 08:17:36 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:17:36 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/04 08:17:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:17:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:17:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4548
2021/12/04 08:17:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:17:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:17:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:17:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (119.36s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/04 08:17:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:18:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:18:05 Rebalance progress: 100
2021/12/04 08:18:10 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 08:18:10 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/04 08:18:10 No eventing-con process running
2021/12/04 08:18:10 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/04 08:18:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:18:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:18:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:18:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:18:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:18:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:18:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:18:24 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:18:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/04 08:18:30 Starting bucket ops to source bucket
2021/12/04 08:18:35 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/04 08:18:40 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/04 08:18:45 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/04 08:18:45 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:18:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:18:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:18:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:18:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:18:50 Rebalance progress: 12.5
2021/12/04 08:18:55 Rebalance progress: 12.5
2021/12/04 08:19:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:19:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:19:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:19:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/04 08:19:00 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/04 08:19:05 Rebalance progress: 12.5
2021/12/04 08:19:10 Rebalance progress: 12.5
2021/12/04 08:19:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:19:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:19:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:19:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:19:15 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/04 08:19:20 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 08:19:25 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 08:19:30 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 08:19:36 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 08:19:36 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/04 08:19:36 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:19:36 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/04 08:19:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:19:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:19:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37555
2021/12/04 08:19:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:19:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:19:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:19:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37555
2021/12/04 08:19:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (124.66s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/04 08:20:09 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 08:20:09 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/04 08:20:09 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:20:09 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/04 08:20:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:20:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:20:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:20:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:20:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:20:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:20:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:20:28 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:20:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:20:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:20:29 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:20:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 08:20:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 08:20:40 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/04 08:20:45 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/04 08:20:50 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/04 08:20:50 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/04 08:20:55 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/04 08:21:00 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/04 08:21:05 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/04 08:21:05 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:21:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:21:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:21:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:21:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:21:11 Rebalance progress: 12.5
2021/12/04 08:21:16 Rebalance progress: 12.5
2021/12/04 08:21:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:21:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:21:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:21:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:21:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:21:26 Rebalance progress: 12.5
2021/12/04 08:21:31 Rebalance progress: 12.5
2021/12/04 08:21:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:21:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:21:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:21:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 08:21:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:21:42 Rebalance progress: 12.5
2021/12/04 08:21:47 Rebalance progress: 12.5
2021/12/04 08:21:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:21:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:21:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:21:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:21:57 Rebalance progress: 12.5
2021/12/04 08:22:02 Rebalance progress: 12.5
2021/12/04 08:22:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:22:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:22:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:22:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 08:22:12 Rebalance progress: 12.5
2021/12/04 08:22:17 Rebalance progress: 12.5
2021/12/04 08:22:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:22:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:22:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:22:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 08:22:27 Rebalance progress: 12.5
2021/12/04 08:22:32 Rebalance progress: 12.5
2021/12/04 08:22:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:22:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:22:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:22:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:22:42 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 08:22:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 08:22:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 08:22:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 08:22:57 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/04 08:22:57 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:22:57 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/04 08:23:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:23:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:23:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2758
2021/12/04 08:23:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2758
2021/12/04 08:23:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:23:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:23:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:23:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12187
2021/12/04 08:23:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:23:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:23:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 08:23:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 08:23:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 08:23:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 08:23:48 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/04 08:23:48 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:23:48 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/04 08:23:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:23:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:23:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:23:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:24:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:24:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (242.27s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/04 08:24:11 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 08:24:11 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/04 08:24:11 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:24:11 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/04 08:24:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:24:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:24:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:24:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:24:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:24:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:24:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:24:30 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:24:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:24:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:24:31 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:24:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 08:24:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 08:24:42 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/04 08:24:47 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/04 08:24:52 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/04 08:24:52 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/04 08:24:57 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/04 08:25:02 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/04 08:25:07 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/04 08:25:07 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:25:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:25:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:25:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:25:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:25:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 08:25:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:25:13 Rebalance progress: 12.5
2021/12/04 08:25:18 Rebalance progress: 12.5
2021/12/04 08:25:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:25:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/04 08:25:28 Rebalance progress: 12.5
2021/12/04 08:25:33 Rebalance progress: 12.5
2021/12/04 08:25:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:25:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:25:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:25:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:25:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 08:25:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 08:25:54 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 08:25:59 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 08:25:59 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/04 08:25:59 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:25:59 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/04 08:26:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:26:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:26:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4431
2021/12/04 08:26:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4431
2021/12/04 08:26:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:26:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:26:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:26:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6101
2021/12/04 08:26:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:26:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:26:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 08:26:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 08:26:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 08:26:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 08:26:49 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/04 08:26:49 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:26:49 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/04 08:26:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:26:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:26:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:26:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:27:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:27:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.39s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/04 08:27:13 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 08:27:13 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/04 08:27:13 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:27:13 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/04 08:27:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:27:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:27:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:27:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:27:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:27:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:27:31 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 08:27:31 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/04 08:27:31 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:27:31 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/04 08:27:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:27:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:27:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:27:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:27:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:27:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:27:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:27:45 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:27:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:27:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:27:46 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:27:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/04 08:27:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 08:27:57 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/04 08:28:02 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/04 08:28:07 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/04 08:28:07 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/04 08:28:12 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/04 08:28:17 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/04 08:28:22 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/04 08:28:22 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:28:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:28:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:28:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:28:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:28:28 Rebalance progress: 12.5
2021/12/04 08:28:33 Rebalance progress: 12.5
2021/12/04 08:28:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:28:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:28:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:28:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:28:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:28:43 Rebalance progress: 12.5
2021/12/04 08:28:48 Rebalance progress: 12.5
2021/12/04 08:28:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:28:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:28:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:28:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 08:28:58 Rebalance progress: 12.5
2021/12/04 08:29:03 Rebalance progress: 12.5
2021/12/04 08:29:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:29:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:29:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:29:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:29:13 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 08:29:18 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 08:29:23 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 08:29:28 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 08:29:28 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/04 08:29:28 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:29:28 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/04 08:29:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:29:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:29:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4922
2021/12/04 08:29:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4922
2021/12/04 08:29:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:29:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:29:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:29:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7599
2021/12/04 08:29:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:29:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:30:04 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 08:30:09 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 08:30:14 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 08:30:19 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 08:30:19 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/04 08:30:19 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:30:19 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/04 08:30:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:30:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:30:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:30:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:30:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:30:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (209.95s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/04 08:30:43 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:30:43 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 08:30:43 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:30:43 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/04 08:30:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:30:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:30:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:30:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:30:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:30:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:31:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:31:01 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:31:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/04 08:31:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 08:31:12 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 08:31:17 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 08:31:22 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 08:31:23 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:31:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:31:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:31:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:31:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:31:43 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:31:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:31:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:32:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:32:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:32:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:32:19 Rebalance progress: 12.5
2021/12/04 08:32:24 Rebalance progress: 12.5
2021/12/04 08:32:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:32:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:32:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:32:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:32:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:32:34 Rebalance progress: 12.5
2021/12/04 08:32:39 Rebalance progress: 12.5
2021/12/04 08:32:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:32:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:32:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:32:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 08:32:49 Rebalance progress: 12.5
2021/12/04 08:32:54 Rebalance progress: 12.5
2021/12/04 08:32:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:32:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:32:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:32:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:33:04 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:33:09 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:33:14 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:33:19 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:33:19 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 08:33:19 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:33:19 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/04 08:33:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:33:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:33:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11155
2021/12/04 08:33:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11155
2021/12/04 08:33:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:33:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:33:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:33:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11155
2021/12/04 08:33:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.14s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/04 08:33:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:33:58 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 08:33:58 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:33:58 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/04 08:34:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:34:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:34:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:34:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:34:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:34:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:34:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:34:17 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:34:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/04 08:34:23 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 08:34:28 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 08:34:33 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 08:34:38 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 08:34:38 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:34:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:34:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:34:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:34:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:34:58 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:34:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:34:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:35:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:35:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:35:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:35:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:35:29 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:35:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:35:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:35:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:35:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:35:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:35:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:36:00 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:36:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:36:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:36:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:36:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:36:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:36:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:36:31 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:36:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:36:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:36:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:36:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:36:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:36:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:37:02 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:37:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:37:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:37:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:37:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:37:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:37:17 Rebalance progress: 12.5
2021/12/04 08:37:22 Rebalance progress: 12.5
2021/12/04 08:37:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:37:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:37:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:37:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:37:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:37:33 Rebalance progress: 12.5
2021/12/04 08:37:38 Rebalance progress: 12.5
2021/12/04 08:37:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:37:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:37:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:37:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 08:37:48 Rebalance progress: 12.5
2021/12/04 08:37:53 Rebalance progress: 12.5
2021/12/04 08:37:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:37:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:37:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:37:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:38:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:38:08 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:38:13 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:38:18 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 08:38:18 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 08:38:18 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:38:18 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/04 08:38:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:38:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:38:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:38:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:38:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:38:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21484
2021/12/04 08:38:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (288.56s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/04 08:38:46 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 08:38:46 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/04 08:38:46 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:38:46 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/04 08:38:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:38:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:38:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:38:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:39:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:39:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:39:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:39:05 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:39:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 08:39:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 08:39:16 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/04 08:39:21 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/04 08:39:26 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/04 08:39:26 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:39:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:39:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:39:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:39:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:39:32 Rebalance progress: 12.5
2021/12/04 08:39:37 Rebalance progress: 12.5
2021/12/04 08:39:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:39:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:39:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:39:42 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:39:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:39:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:39:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:39:57 Rebalance progress: 12.5
2021/12/04 08:40:02 Rebalance progress: 12.5
2021/12/04 08:40:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:40:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:40:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:40:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 08:40:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:40:12 Rebalance progress: 12.5
2021/12/04 08:40:17 Rebalance progress: 12.5
2021/12/04 08:40:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:40:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:40:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:40:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 08:40:27 Rebalance progress: 12.5
2021/12/04 08:40:32 Rebalance progress: 12.5
2021/12/04 08:40:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:40:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:40:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:40:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:40:42 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 08:40:47 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 08:40:52 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 08:40:57 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 08:40:57 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/04 08:40:57 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:40:57 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/04 08:40:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:41:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:41:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8594
2021/12/04 08:41:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:41:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:41:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:41:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8565
2021/12/04 08:41:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8565
2021/12/04 08:41:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.55s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/04 08:41:36 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 08:41:36 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/04 08:41:36 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:41:36 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/04 08:41:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:41:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:41:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:41:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:41:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:41:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:41:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:41:55 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:41:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 08:42:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 08:42:06 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/04 08:42:11 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/04 08:42:16 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/04 08:42:16 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:42:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:42:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:42:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/04 08:42:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:42:21 Rebalance progress: 12.5
2021/12/04 08:42:26 Rebalance progress: 12.5
2021/12/04 08:42:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:42:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:42:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:42:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/04 08:42:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:42:36 Rebalance progress: 12.5
2021/12/04 08:42:41 Rebalance progress: 12.5
2021/12/04 08:42:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:42:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:42:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:42:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 08:42:51 Rebalance progress: 12.5
2021/12/04 08:42:56 Rebalance progress: 12.5
2021/12/04 08:43:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:43:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:43:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:43:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:43:07 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 08:43:12 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 08:43:17 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 08:43:22 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 08:43:22 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/04 08:43:22 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:43:22 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/04 08:43:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:43:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:43:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6061
2021/12/04 08:43:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6061
2021/12/04 08:43:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:43:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:43:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:43:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2633
2021/12/04 08:43:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2633
2021/12/04 08:43:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (149.37s)
=== RUN   TestEventingRebInWithOps
2021/12/04 08:44:05 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 08:44:05 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/04 08:44:05 eventingSetup.go:46: No eventing-con process running
2021/12/04 08:44:05 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/04 08:44:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:44:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:44:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:44:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:44:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:44:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:44:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:44:24 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:44:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 08:44:30 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 08:44:35 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 08:44:40 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 08:44:45 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 08:44:45 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/04 08:44:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:44:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 08:44:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 08:44:51 Rebalance progress: 12.5
2021/12/04 08:44:56 Rebalance progress: 12.5
2021/12/04 08:45:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:45:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:45:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:45:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:45:06 Rebalance progress: 12.5
2021/12/04 08:45:11 Rebalance progress: 12.5
2021/12/04 08:45:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:45:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:45:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:45:16 Waiting for src bucket item count to get to: 200000 curr count: 35407
2021/12/04 08:45:21 Waiting for src bucket item count to get to: 200000 curr count: 51236
2021/12/04 08:45:26 Waiting for src bucket item count to get to: 200000 curr count: 51236
2021/12/04 08:45:31 Waiting for src bucket item count to get to: 200000 curr count: 67253
2021/12/04 08:45:36 Waiting for src bucket item count to get to: 200000 curr count: 67253
2021/12/04 08:45:41 Waiting for src bucket item count to get to: 200000 curr count: 82181
2021/12/04 08:45:46 Waiting for src bucket item count to get to: 200000 curr count: 82181
2021/12/04 08:45:51 Waiting for src bucket item count to get to: 200000 curr count: 97943
2021/12/04 08:45:56 Waiting for src bucket item count to get to: 200000 curr count: 97943
2021/12/04 08:46:01 Waiting for src bucket item count to get to: 200000 curr count: 113558
2021/12/04 08:46:06 Waiting for src bucket item count to get to: 200000 curr count: 113558
2021/12/04 08:46:11 Waiting for src bucket item count to get to: 200000 curr count: 129322
2021/12/04 08:46:16 Waiting for src bucket item count to get to: 200000 curr count: 129322
2021/12/04 08:46:21 Waiting for src bucket item count to get to: 200000 curr count: 145151
2021/12/04 08:46:26 Waiting for src bucket item count to get to: 200000 curr count: 145151
2021/12/04 08:46:31 Waiting for src bucket item count to get to: 200000 curr count: 145151
2021/12/04 08:46:36 Waiting for src bucket item count to get to: 200000 curr count: 161049
2021/12/04 08:46:41 Waiting for src bucket item count to get to: 200000 curr count: 161049
2021/12/04 08:46:46 Waiting for src bucket item count to get to: 200000 curr count: 176334
2021/12/04 08:46:51 Waiting for src bucket item count to get to: 200000 curr count: 176334
2021/12/04 08:46:56 Waiting for src bucket item count to get to: 200000 curr count: 192219
2021/12/04 08:47:01 Waiting for src bucket item count to get to: 200000 curr count: 192219
2021/12/04 08:47:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 08:47:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 08:47:06 Starting bucket ops to source bucket
2021/12/04 08:50:57 Waiting for dst bucket item count to get to: 0 curr count: 5759
2021/12/04 08:51:02 Waiting for dst bucket item count to get to: 0 curr count: 5759
2021/12/04 08:51:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:51:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 08:51:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:51:08 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 08:51:13 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 08:51:18 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 08:51:23 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 08:51:23 App: TestEventingRebInWithOps got un-deployed
2021/12/04 08:51:23 No eventing-con process running
2021/12/04 08:51:23 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/04 08:51:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:51:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:51:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:51:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:51:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:51:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (460.83s)
=== RUN   TestEventingRebOutWithOps
2021/12/04 08:51:41 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 08:51:42 Removing node(s): [] from the cluster
2021/12/04 08:51:47 Rebalance progress: 100
2021/12/04 08:51:52 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 08:51:52 App: TestEventingRebOutWithOps got un-deployed
2021/12/04 08:51:52 No eventing-con process running
2021/12/04 08:51:52 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/04 08:51:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:51:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:51:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:52:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:52:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:52:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:52:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:52:10 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:52:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 08:52:16 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 08:52:21 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 08:52:26 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 08:52:31 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 08:52:31 App: TestEventingRebOutWithOps got deployed
2021/12/04 08:52:31 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 08:52:31 Starting bucket ops to source bucket
2021/12/04 08:52:36 Rebalance progress: 12.5
2021/12/04 08:52:41 Rebalance progress: 12.5
2021/12/04 08:52:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:52:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:52:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 08:52:46 Waiting for src bucket item count to get to: 200000 curr count: 11751
2021/12/04 08:52:51 Waiting for src bucket item count to get to: 200000 curr count: 27549
2021/12/04 08:52:56 Waiting for src bucket item count to get to: 200000 curr count: 27549
2021/12/04 08:53:01 Waiting for src bucket item count to get to: 200000 curr count: 43455
2021/12/04 08:53:06 Waiting for src bucket item count to get to: 200000 curr count: 43455
2021/12/04 08:53:11 Waiting for src bucket item count to get to: 200000 curr count: 59599
2021/12/04 08:53:16 Waiting for src bucket item count to get to: 200000 curr count: 59599
2021/12/04 08:53:21 Waiting for src bucket item count to get to: 200000 curr count: 59599
2021/12/04 08:53:26 Waiting for src bucket item count to get to: 200000 curr count: 74737
2021/12/04 08:53:31 Waiting for src bucket item count to get to: 200000 curr count: 74737
2021/12/04 08:53:36 Waiting for src bucket item count to get to: 200000 curr count: 90276
2021/12/04 08:53:42 Waiting for src bucket item count to get to: 200000 curr count: 90276
2021/12/04 08:53:47 Waiting for src bucket item count to get to: 200000 curr count: 106288
2021/12/04 08:53:52 Waiting for src bucket item count to get to: 200000 curr count: 106288
2021/12/04 08:53:57 Waiting for src bucket item count to get to: 200000 curr count: 122351
2021/12/04 08:54:02 Waiting for src bucket item count to get to: 200000 curr count: 122351
2021/12/04 08:54:07 Waiting for src bucket item count to get to: 200000 curr count: 138647
2021/12/04 08:54:12 Waiting for src bucket item count to get to: 200000 curr count: 138647
2021/12/04 08:54:17 Waiting for src bucket item count to get to: 200000 curr count: 154803
2021/12/04 08:54:22 Waiting for src bucket item count to get to: 200000 curr count: 154803
2021/12/04 08:54:27 Waiting for src bucket item count to get to: 200000 curr count: 169921
2021/12/04 08:54:32 Waiting for src bucket item count to get to: 200000 curr count: 169921
2021/12/04 08:54:37 Waiting for src bucket item count to get to: 200000 curr count: 185666
2021/12/04 08:54:42 Waiting for src bucket item count to get to: 200000 curr count: 185666
2021/12/04 08:54:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 08:54:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 08:54:47 Starting bucket ops to source bucket
2021/12/04 08:58:39 Waiting for dst bucket item count to get to: 0 curr count: 8243
2021/12/04 08:58:44 Waiting for dst bucket item count to get to: 0 curr count: 8243
2021/12/04 08:58:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:58:44 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 08:58:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 08:58:49 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 08:58:54 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 08:58:59 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 08:59:04 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 08:59:04 App: TestEventingRebOutWithOps got un-deployed
2021/12/04 08:59:04 No eventing-con process running
2021/12/04 08:59:04 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/04 08:59:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:59:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:59:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:59:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:59:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:59:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (461.43s)
=== RUN   TestEventingSwapRebWithOps
2021/12/04 08:59:28 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 08:59:28 App: TestEventingSwapRebWithOps got un-deployed
2021/12/04 08:59:28 No eventing-con process running
2021/12/04 08:59:28 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/04 08:59:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:59:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:59:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 08:59:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:59:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:59:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 08:59:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 08:59:46 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 08:59:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 08:59:52 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 08:59:57 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 09:00:02 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 09:00:07 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 09:00:07 App: TestEventingSwapRebWithOps got deployed
2021/12/04 09:00:07 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 09:00:07 Starting bucket ops to source bucket
2021/12/04 09:00:08 Removing node(s): [] from the cluster
2021/12/04 09:00:13 Rebalance progress: 12.5
2021/12/04 09:00:18 Rebalance progress: 12.5
2021/12/04 09:00:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:00:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:00:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:00:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 09:00:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 09:00:29 Rebalance progress: 12.5
2021/12/04 09:00:34 Rebalance progress: 12.5
2021/12/04 09:00:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:00:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:00:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:00:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 09:00:44 Rebalance progress: 12.5
2021/12/04 09:00:49 Rebalance progress: 12.5
2021/12/04 09:00:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:00:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:00:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:00:54 Waiting for src bucket item count to get to: 200000 curr count: 63332
2021/12/04 09:00:59 Waiting for src bucket item count to get to: 200000 curr count: 63332
2021/12/04 09:01:04 Waiting for src bucket item count to get to: 200000 curr count: 78111
2021/12/04 09:01:09 Waiting for src bucket item count to get to: 200000 curr count: 78111
2021/12/04 09:01:14 Waiting for src bucket item count to get to: 200000 curr count: 93851
2021/12/04 09:01:19 Waiting for src bucket item count to get to: 200000 curr count: 93851
2021/12/04 09:01:24 Waiting for src bucket item count to get to: 200000 curr count: 109846
2021/12/04 09:01:29 Waiting for src bucket item count to get to: 200000 curr count: 109846
2021/12/04 09:01:34 Waiting for src bucket item count to get to: 200000 curr count: 125703
2021/12/04 09:01:39 Waiting for src bucket item count to get to: 200000 curr count: 125703
2021/12/04 09:01:44 Waiting for src bucket item count to get to: 200000 curr count: 141804
2021/12/04 09:01:49 Waiting for src bucket item count to get to: 200000 curr count: 141804
2021/12/04 09:01:54 Waiting for src bucket item count to get to: 200000 curr count: 158123
2021/12/04 09:01:59 Waiting for src bucket item count to get to: 200000 curr count: 158123
2021/12/04 09:02:04 Waiting for src bucket item count to get to: 200000 curr count: 173492
2021/12/04 09:02:09 Waiting for src bucket item count to get to: 200000 curr count: 173492
2021/12/04 09:02:14 Waiting for src bucket item count to get to: 200000 curr count: 189498
2021/12/04 09:02:19 Waiting for src bucket item count to get to: 200000 curr count: 189498
2021/12/04 09:02:24 Waiting for src bucket item count to get to: 200000 curr count: 189498
2021/12/04 09:02:29 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 09:02:29 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 09:02:29 Starting bucket ops to source bucket
2021/12/04 09:06:21 Waiting for dst bucket item count to get to: 0 curr count: 11465
2021/12/04 09:06:26 Waiting for dst bucket item count to get to: 0 curr count: 11465
2021/12/04 09:06:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:06:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 09:06:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 09:06:31 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 09:06:36 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 09:06:41 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 09:06:46 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 09:06:46 App: TestEventingSwapRebWithOps got un-deployed
2021/12/04 09:06:46 No eventing-con process running
2021/12/04 09:06:46 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/04 09:06:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:06:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:06:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:06:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:06:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:06:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (461.79s)
=== RUN   TestKVRebInWithOps
2021/12/04 09:07:09 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 09:07:09 App: TestKVRebInWithOps got un-deployed
2021/12/04 09:07:09 No eventing-con process running
2021/12/04 09:07:09 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/04 09:07:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:07:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:07:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:07:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:07:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:07:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:07:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:07:28 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 09:07:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 09:07:34 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 09:07:39 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 09:07:44 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 09:07:49 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 09:07:49 App: TestKVRebInWithOps got deployed
2021/12/04 09:07:49 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 09:07:49 Starting bucket ops to source bucket
2021/12/04 09:07:49 Removing node(s): [] from the cluster
2021/12/04 09:07:54 Rebalance progress: 12.5
2021/12/04 09:07:59 Rebalance progress: 12.5
2021/12/04 09:08:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:08:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:08:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:08:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 09:08:10 Rebalance progress: 12.5
2021/12/04 09:08:15 Rebalance progress: 12.5
2021/12/04 09:08:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:08:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:08:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:08:20 Waiting for src bucket item count to get to: 200000 curr count: 29800
2021/12/04 09:08:25 Waiting for src bucket item count to get to: 200000 curr count: 45938
2021/12/04 09:08:30 Waiting for src bucket item count to get to: 200000 curr count: 45938
2021/12/04 09:08:35 Waiting for src bucket item count to get to: 200000 curr count: 61720
2021/12/04 09:08:40 Waiting for src bucket item count to get to: 200000 curr count: 61720
2021/12/04 09:08:45 Waiting for src bucket item count to get to: 200000 curr count: 61720
2021/12/04 09:08:50 Waiting for src bucket item count to get to: 200000 curr count: 76597
2021/12/04 09:08:55 Waiting for src bucket item count to get to: 200000 curr count: 92371
2021/12/04 09:09:00 Waiting for src bucket item count to get to: 200000 curr count: 92371
2021/12/04 09:09:05 Waiting for src bucket item count to get to: 200000 curr count: 108279
2021/12/04 09:09:10 Waiting for src bucket item count to get to: 200000 curr count: 108279
2021/12/04 09:09:15 Waiting for src bucket item count to get to: 200000 curr count: 124315
2021/12/04 09:09:20 Waiting for src bucket item count to get to: 200000 curr count: 124315
2021/12/04 09:09:25 Waiting for src bucket item count to get to: 200000 curr count: 139973
2021/12/04 09:09:30 Waiting for src bucket item count to get to: 200000 curr count: 139973
2021/12/04 09:09:35 Waiting for src bucket item count to get to: 200000 curr count: 155991
2021/12/04 09:09:40 Waiting for src bucket item count to get to: 200000 curr count: 155991
2021/12/04 09:09:45 Waiting for src bucket item count to get to: 200000 curr count: 171039
2021/12/04 09:09:50 Waiting for src bucket item count to get to: 200000 curr count: 171039
2021/12/04 09:09:55 Waiting for src bucket item count to get to: 200000 curr count: 186793
2021/12/04 09:10:00 Waiting for src bucket item count to get to: 200000 curr count: 186793
2021/12/04 09:10:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 09:10:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 09:10:05 Starting bucket ops to source bucket
2021/12/04 09:13:57 Waiting for dst bucket item count to get to: 0 curr count: 5030
2021/12/04 09:14:02 Waiting for dst bucket item count to get to: 0 curr count: 5030
2021/12/04 09:14:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:14:02 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 09:14:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 09:14:07 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 09:14:12 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 09:14:17 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 09:14:22 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 09:14:22 App: TestKVRebInWithOps got un-deployed
2021/12/04 09:14:22 No eventing-con process running
2021/12/04 09:14:22 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/04 09:14:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:14:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:14:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:14:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:14:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:14:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (456.40s)
=== RUN   TestKVRebOutWithOps
2021/12/04 09:14:41 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 09:14:41 Removing node(s): [] from the cluster
2021/12/04 09:14:46 Rebalance progress: 100
2021/12/04 09:14:51 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 09:14:51 App: TestKVRebOutWithOps got un-deployed
2021/12/04 09:14:51 No eventing-con process running
2021/12/04 09:14:51 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/04 09:14:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:14:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:14:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:15:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:15:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:15:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:15:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:15:10 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 09:15:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 09:15:16 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 09:15:21 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 09:15:26 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 09:15:31 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 09:15:31 App: TestKVRebOutWithOps got deployed
2021/12/04 09:15:31 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 09:15:31 Starting bucket ops to source bucket
2021/12/04 09:15:36 Rebalance progress: 12.5
2021/12/04 09:15:41 Rebalance progress: 12.5
2021/12/04 09:15:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:15:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:15:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:15:46 Waiting for src bucket item count to get to: 200000 curr count: 12597
2021/12/04 09:15:51 Waiting for src bucket item count to get to: 200000 curr count: 28264
2021/12/04 09:15:56 Waiting for src bucket item count to get to: 200000 curr count: 28264
2021/12/04 09:16:01 Waiting for src bucket item count to get to: 200000 curr count: 44327
2021/12/04 09:16:06 Waiting for src bucket item count to get to: 200000 curr count: 44327
2021/12/04 09:16:11 Waiting for src bucket item count to get to: 200000 curr count: 60330
2021/12/04 09:16:16 Waiting for src bucket item count to get to: 200000 curr count: 60330
2021/12/04 09:16:21 Waiting for src bucket item count to get to: 200000 curr count: 60330
2021/12/04 09:16:26 Waiting for src bucket item count to get to: 200000 curr count: 75325
2021/12/04 09:16:31 Waiting for src bucket item count to get to: 200000 curr count: 75325
2021/12/04 09:16:36 Waiting for src bucket item count to get to: 200000 curr count: 90889
2021/12/04 09:16:41 Waiting for src bucket item count to get to: 200000 curr count: 90889
2021/12/04 09:16:46 Waiting for src bucket item count to get to: 200000 curr count: 106793
2021/12/04 09:16:51 Waiting for src bucket item count to get to: 200000 curr count: 106793
2021/12/04 09:16:56 Waiting for src bucket item count to get to: 200000 curr count: 122643
2021/12/04 09:17:01 Waiting for src bucket item count to get to: 200000 curr count: 122643
2021/12/04 09:17:06 Waiting for src bucket item count to get to: 200000 curr count: 138371
2021/12/04 09:17:11 Waiting for src bucket item count to get to: 200000 curr count: 138371
2021/12/04 09:17:16 Waiting for src bucket item count to get to: 200000 curr count: 154249
2021/12/04 09:17:21 Waiting for src bucket item count to get to: 200000 curr count: 154249
2021/12/04 09:17:26 Waiting for src bucket item count to get to: 200000 curr count: 169289
2021/12/04 09:17:31 Waiting for src bucket item count to get to: 200000 curr count: 169289
2021/12/04 09:17:36 Waiting for src bucket item count to get to: 200000 curr count: 185133
2021/12/04 09:17:41 Waiting for src bucket item count to get to: 200000 curr count: 185133
2021/12/04 09:17:46 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 09:17:46 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 09:17:46 Starting bucket ops to source bucket
2021/12/04 09:21:38 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/04 09:21:43 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/04 09:21:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:21:43 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 09:21:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 09:21:48 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 09:21:53 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 09:21:58 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 09:22:03 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 09:22:03 App: TestKVRebOutWithOps got un-deployed
2021/12/04 09:22:03 No eventing-con process running
2021/12/04 09:22:03 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/04 09:22:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:22:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:22:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:22:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:22:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:22:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (461.13s)
=== RUN   TestKVSwapRebWithOps
2021/12/04 09:22:27 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 09:22:27 App: TestKVSwapRebWithOps got un-deployed
2021/12/04 09:22:27 No eventing-con process running
2021/12/04 09:22:27 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/04 09:22:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:22:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:22:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:22:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:22:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:22:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:22:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:22:46 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 09:22:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 09:22:52 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 09:22:57 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 09:23:02 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 09:23:07 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 09:23:07 App: TestKVSwapRebWithOps got deployed
2021/12/04 09:23:07 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 09:23:07 Starting bucket ops to source bucket
2021/12/04 09:23:07 Removing node(s): [] from the cluster
2021/12/04 09:23:12 Rebalance progress: 12.5
2021/12/04 09:23:17 Rebalance progress: 12.5
2021/12/04 09:23:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:23:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:23:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:23:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 09:23:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 09:23:27 Rebalance progress: 12.5
2021/12/04 09:23:32 Rebalance progress: 12.5
2021/12/04 09:23:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:23:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:23:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:23:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 09:23:42 Rebalance progress: 12.5
2021/12/04 09:23:47 Rebalance progress: 12.5
2021/12/04 09:23:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:23:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:23:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:23:53 Waiting for src bucket item count to get to: 200000 curr count: 64418
2021/12/04 09:23:58 Waiting for src bucket item count to get to: 200000 curr count: 64418
2021/12/04 09:24:03 Waiting for src bucket item count to get to: 200000 curr count: 78927
2021/12/04 09:24:08 Waiting for src bucket item count to get to: 200000 curr count: 78927
2021/12/04 09:24:13 Waiting for src bucket item count to get to: 200000 curr count: 94732
2021/12/04 09:24:18 Waiting for src bucket item count to get to: 200000 curr count: 94732
2021/12/04 09:24:23 Waiting for src bucket item count to get to: 200000 curr count: 110709
2021/12/04 09:24:28 Waiting for src bucket item count to get to: 200000 curr count: 110709
2021/12/04 09:24:33 Waiting for src bucket item count to get to: 200000 curr count: 126815
2021/12/04 09:24:38 Waiting for src bucket item count to get to: 200000 curr count: 126815
2021/12/04 09:24:43 Waiting for src bucket item count to get to: 200000 curr count: 142985
2021/12/04 09:24:48 Waiting for src bucket item count to get to: 200000 curr count: 142985
2021/12/04 09:24:53 Waiting for src bucket item count to get to: 200000 curr count: 142985
2021/12/04 09:24:58 Waiting for src bucket item count to get to: 200000 curr count: 159113
2021/12/04 09:25:03 Waiting for src bucket item count to get to: 200000 curr count: 159113
2021/12/04 09:25:08 Waiting for src bucket item count to get to: 200000 curr count: 174322
2021/12/04 09:25:13 Waiting for src bucket item count to get to: 200000 curr count: 174322
2021/12/04 09:25:18 Waiting for src bucket item count to get to: 200000 curr count: 190067
2021/12/04 09:25:23 Waiting for src bucket item count to get to: 200000 curr count: 190067
2021/12/04 09:25:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 09:25:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 09:25:28 Starting bucket ops to source bucket
2021/12/04 09:29:20 Waiting for dst bucket item count to get to: 0 curr count: 10039
2021/12/04 09:29:25 Waiting for dst bucket item count to get to: 0 curr count: 10039
2021/12/04 09:29:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:29:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 09:29:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 09:29:30 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 09:29:35 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 09:29:40 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 09:29:45 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 09:29:45 App: TestKVSwapRebWithOps got un-deployed
2021/12/04 09:29:45 No eventing-con process running
2021/12/04 09:29:45 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/04 09:29:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:29:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:29:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:29:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:29:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:29:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (461.62s)
=== RUN   TestEventingRebInWithTimers
2021/12/04 09:30:09 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 09:30:09 App: TestEventingRebInWithTimers got un-deployed
2021/12/04 09:30:09 No eventing-con process running
2021/12/04 09:30:09 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/04 09:30:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:30:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:30:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:30:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:30:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:30:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:30:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:30:27 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 09:30:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 09:30:33 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 09:30:38 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 09:30:43 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 09:30:48 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 09:30:48 App: TestEventingRebInWithTimers got deployed
2021/12/04 09:30:48 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 09:30:48 Starting bucket ops to source bucket
2021/12/04 09:30:49 Removing node(s): [] from the cluster
2021/12/04 09:30:54 Rebalance progress: 12.5
2021/12/04 09:30:59 Rebalance progress: 12.5
2021/12/04 09:31:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:31:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:31:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:31:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 09:31:09 Rebalance progress: 12.5
2021/12/04 09:31:14 Rebalance progress: 12.5
2021/12/04 09:31:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:31:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:31:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:31:19 Waiting for src bucket item count to get to: 50000 curr count: 27806
2021/12/04 09:31:24 Waiting for src bucket item count to get to: 50000 curr count: 37253
2021/12/04 09:31:29 Waiting for src bucket item count to get to: 50000 curr count: 37253
2021/12/04 09:31:39 Waiting for dst bucket item count to get to: 50000 curr count: 27337
2021/12/04 09:31:44 Waiting for dst bucket item count to get to: 50000 curr count: 27337
2021/12/04 09:31:49 Waiting for dst bucket item count to get to: 50000 curr count: 44828
2021/12/04 09:31:54 Waiting for dst bucket item count to get to: 50000 curr count: 44828
2021/12/04 09:31:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 09:31:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 09:31:54 Starting bucket ops to source bucket
2021/12/04 09:33:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:33:10 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:33:15 Waiting for dst bucket item count to get to: 0 curr count: 43329
2021/12/04 09:33:20 Waiting for dst bucket item count to get to: 0 curr count: 43329
2021/12/04 09:33:25 Waiting for dst bucket item count to get to: 0 curr count: 32462
2021/12/04 09:33:30 Waiting for dst bucket item count to get to: 0 curr count: 32462
2021/12/04 09:33:35 Waiting for dst bucket item count to get to: 0 curr count: 19006
2021/12/04 09:33:40 Waiting for dst bucket item count to get to: 0 curr count: 19006
2021/12/04 09:33:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:33:40 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 09:33:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 09:33:46 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 09:33:51 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 09:33:56 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 09:34:01 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 09:34:01 App: TestEventingRebInWithTimers got un-deployed
2021/12/04 09:34:01 No eventing-con process running
2021/12/04 09:34:01 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/04 09:34:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:34:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:34:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:34:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:34:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:34:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (255.68s)
=== RUN   TestEventingRebOutWithTimers
2021/12/04 09:34:19 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 09:34:19 Removing node(s): [] from the cluster
2021/12/04 09:34:24 Rebalance progress: 100
2021/12/04 09:34:29 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 09:34:29 App: TestEventingRebOutWithTimers got un-deployed
2021/12/04 09:34:29 No eventing-con process running
2021/12/04 09:34:29 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/04 09:34:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:34:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:34:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:34:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:34:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:34:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:34:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:34:48 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 09:34:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 09:34:54 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 09:34:59 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 09:35:04 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 09:35:09 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 09:35:09 App: TestEventingRebOutWithTimers got deployed
2021/12/04 09:35:09 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 09:35:09 Starting bucket ops to source bucket
2021/12/04 09:35:14 Rebalance progress: 12.5
2021/12/04 09:35:19 Rebalance progress: 12.5
2021/12/04 09:35:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:35:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:35:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:35:24 Waiting for src bucket item count to get to: 50000 curr count: 13863
2021/12/04 09:35:29 Waiting for src bucket item count to get to: 50000 curr count: 13863
2021/12/04 09:35:34 Waiting for src bucket item count to get to: 50000 curr count: 25908
2021/12/04 09:35:39 Waiting for src bucket item count to get to: 50000 curr count: 25908
2021/12/04 09:35:44 Waiting for src bucket item count to get to: 50000 curr count: 35527
2021/12/04 09:35:49 Waiting for src bucket item count to get to: 50000 curr count: 35527
2021/12/04 09:35:54 Waiting for src bucket item count to get to: 50000 curr count: 35527
2021/12/04 09:35:59 Waiting for src bucket item count to get to: 50000 curr count: 47655
2021/12/04 09:36:04 Waiting for src bucket item count to get to: 50000 curr count: 47655
2021/12/04 09:36:14 Waiting for dst bucket item count to get to: 50000 curr count: 47131
2021/12/04 09:36:19 Waiting for dst bucket item count to get to: 50000 curr count: 49920
2021/12/04 09:36:24 Waiting for dst bucket item count to get to: 50000 curr count: 49920
2021/12/04 09:36:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 09:36:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 09:36:24 Starting bucket ops to source bucket
2021/12/04 09:37:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:37:40 Waiting for dst bucket item count to get to: 0 curr count: 49060
2021/12/04 09:37:45 Waiting for dst bucket item count to get to: 0 curr count: 49060
2021/12/04 09:37:50 Waiting for dst bucket item count to get to: 0 curr count: 40186
2021/12/04 09:37:55 Waiting for dst bucket item count to get to: 0 curr count: 40186
2021/12/04 09:38:00 Waiting for dst bucket item count to get to: 0 curr count: 29341
2021/12/04 09:38:05 Waiting for dst bucket item count to get to: 0 curr count: 29341
2021/12/04 09:38:10 Waiting for dst bucket item count to get to: 0 curr count: 6602
2021/12/04 09:38:15 Waiting for dst bucket item count to get to: 0 curr count: 6602
2021/12/04 09:38:20 Waiting for dst bucket item count to get to: 0 curr count: 646
2021/12/04 09:38:25 Waiting for dst bucket item count to get to: 0 curr count: 646
2021/12/04 09:38:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:38:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 09:38:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 09:38:30 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 09:38:35 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 09:38:40 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 09:38:45 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 09:38:45 App: TestEventingRebOutWithTimers got un-deployed
2021/12/04 09:38:45 No eventing-con process running
2021/12/04 09:38:45 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/04 09:38:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:38:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:38:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:38:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:38:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:38:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (284.41s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/04 09:39:09 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 09:39:09 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/04 09:39:09 No eventing-con process running
2021/12/04 09:39:09 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/04 09:39:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:39:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:39:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:39:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:39:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:39:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:39:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:39:27 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 09:39:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 09:39:33 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 09:39:38 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 09:39:43 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 09:39:48 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 09:39:48 App: TestEventingSwapRebWithTimers got deployed
2021/12/04 09:39:48 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 09:39:48 Starting bucket ops to source bucket
2021/12/04 09:39:49 Removing node(s): [] from the cluster
2021/12/04 09:39:54 Rebalance progress: 12.5
2021/12/04 09:39:59 Rebalance progress: 12.5
2021/12/04 09:40:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:40:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:40:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:40:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 09:40:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 09:40:09 Rebalance progress: 12.5
2021/12/04 09:40:14 Rebalance progress: 12.5
2021/12/04 09:40:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:40:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:40:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:40:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 09:40:25 Rebalance progress: 12.5
2021/12/04 09:40:30 Rebalance progress: 12.5
2021/12/04 09:40:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:40:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:40:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:40:35 Waiting for src bucket item count to get to: 50000 curr count: 47804
2021/12/04 09:40:40 Waiting for src bucket item count to get to: 50000 curr count: 47804
2021/12/04 09:40:50 Waiting for dst bucket item count to get to: 50000 curr count: 40485
2021/12/04 09:40:55 Waiting for dst bucket item count to get to: 50000 curr count: 40485
2021/12/04 09:41:00 Waiting for dst bucket item count to get to: 50000 curr count: 48453
2021/12/04 09:41:05 Waiting for dst bucket item count to get to: 50000 curr count: 48453
2021/12/04 09:41:05 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 09:41:05 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 09:41:05 Starting bucket ops to source bucket
2021/12/04 09:42:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:42:20 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:42:25 Waiting for dst bucket item count to get to: 0 curr count: 47641
2021/12/04 09:42:30 Waiting for dst bucket item count to get to: 0 curr count: 47641
2021/12/04 09:42:35 Waiting for dst bucket item count to get to: 0 curr count: 47641
2021/12/04 09:42:40 Waiting for dst bucket item count to get to: 0 curr count: 35215
2021/12/04 09:42:45 Waiting for dst bucket item count to get to: 0 curr count: 35215
2021/12/04 09:42:50 Waiting for dst bucket item count to get to: 0 curr count: 16374
2021/12/04 09:42:55 Waiting for dst bucket item count to get to: 0 curr count: 16374
2021/12/04 09:43:00 Waiting for dst bucket item count to get to: 0 curr count: 4616
2021/12/04 09:43:05 Waiting for dst bucket item count to get to: 0 curr count: 4616
2021/12/04 09:43:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:43:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 09:43:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 09:43:11 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 09:43:16 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 09:43:21 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 09:43:26 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 09:43:26 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/04 09:43:26 No eventing-con process running
2021/12/04 09:43:26 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/04 09:43:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:43:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:43:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:43:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:43:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:43:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (280.61s)
=== RUN   TestKVRebInWithTimers
2021/12/04 09:43:49 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 09:43:49 App: TestKVRebInWithTimers got un-deployed
2021/12/04 09:43:49 No eventing-con process running
2021/12/04 09:43:49 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/04 09:43:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:43:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:43:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:43:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:44:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:44:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:44:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:44:08 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 09:44:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 09:44:14 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 09:44:19 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 09:44:24 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 09:44:29 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 09:44:29 App: TestKVRebInWithTimers got deployed
2021/12/04 09:44:29 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 09:44:29 Starting bucket ops to source bucket
2021/12/04 09:44:29 Removing node(s): [] from the cluster
2021/12/04 09:44:34 Rebalance progress: 12.5
2021/12/04 09:44:39 Rebalance progress: 12.5
2021/12/04 09:44:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:44:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:44:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:44:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 09:44:50 Rebalance progress: 12.5
2021/12/04 09:44:55 Rebalance progress: 12.5
2021/12/04 09:45:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:45:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:45:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:45:00 Waiting for src bucket item count to get to: 50000 curr count: 26123
2021/12/04 09:45:05 Waiting for src bucket item count to get to: 50000 curr count: 35520
2021/12/04 09:45:10 Waiting for src bucket item count to get to: 50000 curr count: 35520
2021/12/04 09:45:15 Waiting for src bucket item count to get to: 50000 curr count: 47457
2021/12/04 09:45:20 Waiting for src bucket item count to get to: 50000 curr count: 47457
2021/12/04 09:45:30 Waiting for dst bucket item count to get to: 50000 curr count: 42670
2021/12/04 09:45:35 Waiting for dst bucket item count to get to: 50000 curr count: 42670
2021/12/04 09:45:40 Waiting for dst bucket item count to get to: 50000 curr count: 49264
2021/12/04 09:45:45 Waiting for dst bucket item count to get to: 50000 curr count: 49264
2021/12/04 09:45:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 09:45:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 09:45:45 Starting bucket ops to source bucket
2021/12/04 09:46:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:47:00 Waiting for dst bucket item count to get to: 0 curr count: 49544
2021/12/04 09:47:05 Waiting for dst bucket item count to get to: 0 curr count: 49544
2021/12/04 09:47:10 Waiting for dst bucket item count to get to: 0 curr count: 45714
2021/12/04 09:47:15 Waiting for dst bucket item count to get to: 0 curr count: 45714
2021/12/04 09:47:20 Waiting for dst bucket item count to get to: 0 curr count: 35537
2021/12/04 09:47:25 Waiting for dst bucket item count to get to: 0 curr count: 35537
2021/12/04 09:47:30 Waiting for dst bucket item count to get to: 0 curr count: 13794
2021/12/04 09:47:35 Waiting for dst bucket item count to get to: 0 curr count: 13794
2021/12/04 09:47:40 Waiting for dst bucket item count to get to: 0 curr count: 3649
2021/12/04 09:47:45 Waiting for dst bucket item count to get to: 0 curr count: 3649
2021/12/04 09:47:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:47:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 09:47:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 09:47:51 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 09:47:56 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 09:48:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 09:48:06 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 09:48:06 App: TestKVRebInWithTimers got un-deployed
2021/12/04 09:48:06 No eventing-con process running
2021/12/04 09:48:06 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/04 09:48:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:48:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:48:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:48:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:48:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:48:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (280.09s)
=== RUN   TestKVRebOutWithTimers
2021/12/04 09:48:24 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 09:48:25 Removing node(s): [] from the cluster
2021/12/04 09:48:30 Rebalance progress: 100
2021/12/04 09:48:35 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 09:48:35 App: TestKVRebOutWithTimers got un-deployed
2021/12/04 09:48:35 No eventing-con process running
2021/12/04 09:48:35 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/04 09:48:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:48:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:48:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:48:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:48:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:48:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:48:53 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 09:48:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 09:48:59 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 09:49:04 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 09:49:09 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 09:49:14 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 09:49:14 App: TestKVRebOutWithTimers got deployed
2021/12/04 09:49:14 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 09:49:14 Starting bucket ops to source bucket
2021/12/04 09:49:19 Rebalance progress: 12.5
2021/12/04 09:49:24 Rebalance progress: 12.5
2021/12/04 09:49:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:49:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:49:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:49:29 Waiting for src bucket item count to get to: 50000 curr count: 6325
2021/12/04 09:49:34 Waiting for src bucket item count to get to: 50000 curr count: 19914
2021/12/04 09:49:39 Waiting for src bucket item count to get to: 50000 curr count: 19914
2021/12/04 09:49:44 Waiting for src bucket item count to get to: 50000 curr count: 31160
2021/12/04 09:49:49 Waiting for src bucket item count to get to: 50000 curr count: 31160
2021/12/04 09:49:55 Waiting for src bucket item count to get to: 50000 curr count: 31160
2021/12/04 09:50:00 Waiting for src bucket item count to get to: 50000 curr count: 42737
2021/12/04 09:50:10 Waiting for dst bucket item count to get to: 50000 curr count: 34728
2021/12/04 09:50:15 Waiting for dst bucket item count to get to: 50000 curr count: 34728
2021/12/04 09:50:20 Waiting for dst bucket item count to get to: 50000 curr count: 44475
2021/12/04 09:50:25 Waiting for dst bucket item count to get to: 50000 curr count: 44475
2021/12/04 09:50:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 09:50:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 09:50:25 Starting bucket ops to source bucket
2021/12/04 09:51:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:51:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:51:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:51:50 Waiting for dst bucket item count to get to: 0 curr count: 48264
2021/12/04 09:51:55 Waiting for dst bucket item count to get to: 0 curr count: 48264
2021/12/04 09:52:00 Waiting for dst bucket item count to get to: 0 curr count: 31706
2021/12/04 09:52:05 Waiting for dst bucket item count to get to: 0 curr count: 31706
2021/12/04 09:52:10 Waiting for dst bucket item count to get to: 0 curr count: 16319
2021/12/04 09:52:15 Waiting for dst bucket item count to get to: 0 curr count: 16319
2021/12/04 09:52:20 Waiting for dst bucket item count to get to: 0 curr count: 1935
2021/12/04 09:52:25 Waiting for dst bucket item count to get to: 0 curr count: 1935
2021/12/04 09:52:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:52:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 09:52:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 09:52:30 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 09:52:35 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 09:52:40 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 09:52:45 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 09:52:45 App: TestKVRebOutWithTimers got un-deployed
2021/12/04 09:52:45 No eventing-con process running
2021/12/04 09:52:45 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/04 09:52:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:52:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:52:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:52:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:52:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:52:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (279.71s)
=== RUN   TestKVSwapRebWithTimers
2021/12/04 09:53:09 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 09:53:09 App: TestKVSwapRebWithTimers got un-deployed
2021/12/04 09:53:09 No eventing-con process running
2021/12/04 09:53:09 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/04 09:53:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:53:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:53:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:53:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:53:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:53:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:53:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:53:28 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 09:53:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 09:53:34 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 09:53:39 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 09:53:44 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 09:53:49 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 09:53:49 App: TestKVSwapRebWithTimers got deployed
2021/12/04 09:53:49 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 09:53:49 Starting bucket ops to source bucket
2021/12/04 09:53:50 Removing node(s): [] from the cluster
2021/12/04 09:53:55 Rebalance progress: 12.5
2021/12/04 09:54:00 Rebalance progress: 12.5
2021/12/04 09:54:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:54:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:54:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:54:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 09:54:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 09:54:11 Rebalance progress: 12.5
2021/12/04 09:54:16 Rebalance progress: 12.5
2021/12/04 09:54:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:54:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:54:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:54:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 09:54:26 Rebalance progress: 12.5
2021/12/04 09:54:31 Rebalance progress: 12.5
2021/12/04 09:54:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:54:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:54:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:54:36 Waiting for src bucket item count to get to: 50000 curr count: 47674
2021/12/04 09:54:41 Waiting for src bucket item count to get to: 50000 curr count: 47674
2021/12/04 09:54:51 Waiting for dst bucket item count to get to: 50000 curr count: 40931
2021/12/04 09:54:56 Waiting for dst bucket item count to get to: 50000 curr count: 40931
2021/12/04 09:55:01 Waiting for dst bucket item count to get to: 50000 curr count: 48954
2021/12/04 09:55:06 Waiting for dst bucket item count to get to: 50000 curr count: 48954
2021/12/04 09:55:06 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 09:55:06 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 09:55:06 Starting bucket ops to source bucket
2021/12/04 09:56:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:56:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:56:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:56:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 09:56:38 Waiting for dst bucket item count to get to: 0 curr count: 40258
2021/12/04 09:56:43 Waiting for dst bucket item count to get to: 0 curr count: 40258
2021/12/04 09:56:48 Waiting for dst bucket item count to get to: 0 curr count: 20158
2021/12/04 09:56:53 Waiting for dst bucket item count to get to: 0 curr count: 20158
2021/12/04 09:56:58 Waiting for dst bucket item count to get to: 0 curr count: 8546
2021/12/04 09:57:03 Waiting for dst bucket item count to get to: 0 curr count: 8546
2021/12/04 09:57:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:57:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 09:57:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 09:57:08 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 09:57:13 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 09:57:18 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 09:57:23 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 09:57:28 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 09:57:33 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 09:57:33 App: TestKVSwapRebWithTimers got un-deployed
2021/12/04 09:57:33 No eventing-con process running
2021/12/04 09:57:33 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/04 09:57:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:57:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:57:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:57:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:57:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:57:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (287.62s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/04 09:57:57 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 09:57:57 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/04 09:57:57 No eventing-con process running
2021/12/04 09:57:57 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/04 09:57:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:58:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:58:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 09:58:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:58:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:58:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:58:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 09:58:15 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 09:58:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 09:58:21 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 09:58:26 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 09:58:32 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 09:58:37 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 09:58:37 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/04 09:58:37 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 09:58:37 Starting bucket ops to source bucket
2021/12/04 09:58:37 Removing node(s): [] from the cluster
2021/12/04 09:58:42 Rebalance progress: 12.5
2021/12/04 09:58:47 Rebalance progress: 12.5
2021/12/04 09:58:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:58:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:58:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:58:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 09:58:57 Rebalance progress: 12.5
2021/12/04 09:59:02 Rebalance progress: 12.5
2021/12/04 09:59:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:59:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 09:59:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 09:59:07 Waiting for src bucket item count to get to: 50000 curr count: 28218
2021/12/04 09:59:12 Waiting for src bucket item count to get to: 50000 curr count: 38537
2021/12/04 09:59:17 Waiting for src bucket item count to get to: 50000 curr count: 38537
2021/12/04 09:59:22 Waiting for src bucket item count to get to: 50000 curr count: 48975
2021/12/04 09:59:27 Waiting for src bucket item count to get to: 50000 curr count: 48975
2021/12/04 09:59:37 Waiting for dst bucket item count to get to: 50000 curr count: 40199
2021/12/04 09:59:42 Waiting for dst bucket item count to get to: 50000 curr count: 40199
2021/12/04 09:59:47 Waiting for dst bucket item count to get to: 50000 curr count: 49474
2021/12/04 09:59:52 Waiting for dst bucket item count to get to: 50000 curr count: 49474
2021/12/04 09:59:52 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 09:59:52 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 09:59:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 09:59:57 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 10:00:02 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 10:00:07 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 10:00:12 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 10:00:17 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 10:00:22 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 10:00:22 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/04 10:00:22 No eventing-con process running
2021/12/04 10:00:22 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/04 10:00:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:00:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:00:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:00:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:00:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:00:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:00:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:00:51 Waiting for dst bucket item count to get to: 0 curr count: 49782
2021/12/04 10:00:56 Waiting for dst bucket item count to get to: 0 curr count: 49782
2021/12/04 10:00:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (189.40s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/04 10:01:01 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 10:01:01 Removing node(s): [] from the cluster
2021/12/04 10:01:06 Rebalance progress: 100
2021/12/04 10:01:11 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 10:01:11 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/04 10:01:11 No eventing-con process running
2021/12/04 10:01:11 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/04 10:01:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:01:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:01:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:01:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:01:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:01:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:01:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:01:30 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:01:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 10:01:36 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 10:01:41 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 10:01:46 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 10:01:51 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 10:01:51 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/04 10:01:51 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:01:51 Starting bucket ops to source bucket
2021/12/04 10:01:56 Rebalance progress: 12.5
2021/12/04 10:02:01 Rebalance progress: 12.5
2021/12/04 10:02:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:02:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:02:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:02:06 Waiting for src bucket item count to get to: 50000 curr count: 10201
2021/12/04 10:02:11 Waiting for src bucket item count to get to: 50000 curr count: 23854
2021/12/04 10:02:16 Waiting for src bucket item count to get to: 50000 curr count: 23854
2021/12/04 10:02:21 Waiting for src bucket item count to get to: 50000 curr count: 37261
2021/12/04 10:02:26 Waiting for src bucket item count to get to: 50000 curr count: 37261
2021/12/04 10:02:31 Waiting for src bucket item count to get to: 50000 curr count: 46561
2021/12/04 10:02:36 Waiting for src bucket item count to get to: 50000 curr count: 46561
2021/12/04 10:02:41 Waiting for src bucket item count to get to: 50000 curr count: 46561
2021/12/04 10:02:51 Waiting for dst bucket item count to get to: 50000 curr count: 28245
2021/12/04 10:02:56 Waiting for dst bucket item count to get to: 50000 curr count: 28245
2021/12/04 10:03:01 Waiting for dst bucket item count to get to: 50000 curr count: 43824
2021/12/04 10:03:06 Waiting for dst bucket item count to get to: 50000 curr count: 43824
2021/12/04 10:03:06 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 10:03:06 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 10:03:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:03:12 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 10:03:17 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 10:03:22 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 10:03:27 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 10:03:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 10:03:37 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 10:03:37 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/04 10:03:37 No eventing-con process running
2021/12/04 10:03:37 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/04 10:03:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:03:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:03:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:03:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:03:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:03:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:04:01 Waiting for dst bucket item count to get to: 0 curr count: 31591
2021/12/04 10:04:06 Waiting for dst bucket item count to get to: 0 curr count: 31591
2021/12/04 10:04:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.48s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/04 10:04:16 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 10:04:16 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/04 10:04:16 No eventing-con process running
2021/12/04 10:04:16 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/04 10:04:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:04:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:04:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:04:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:04:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:04:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:04:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:04:34 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:04:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 10:04:40 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 10:04:45 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 10:04:50 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 10:04:55 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 10:04:55 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/04 10:04:55 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 10:04:55 Starting bucket ops to source bucket
2021/12/04 10:04:56 Removing node(s): [] from the cluster
2021/12/04 10:05:01 Rebalance progress: 12.5
2021/12/04 10:05:06 Rebalance progress: 12.5
2021/12/04 10:05:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:05:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:05:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:05:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 10:05:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:05:16 Rebalance progress: 12.5
2021/12/04 10:05:21 Rebalance progress: 12.5
2021/12/04 10:05:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:05:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:05:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:05:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 10:05:31 Rebalance progress: 12.5
2021/12/04 10:05:36 Rebalance progress: 12.5
2021/12/04 10:05:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:05:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:05:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:05:41 Waiting for src bucket item count to get to: 50000 curr count: 49158
2021/12/04 10:05:46 Waiting for src bucket item count to get to: 50000 curr count: 49158
2021/12/04 10:05:56 Waiting for dst bucket item count to get to: 50000 curr count: 38621
2021/12/04 10:06:01 Waiting for dst bucket item count to get to: 50000 curr count: 38621
2021/12/04 10:06:01 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 10:06:01 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 10:06:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:06:07 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 10:06:12 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 10:06:17 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 10:06:22 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 10:06:27 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 10:06:27 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/04 10:06:27 No eventing-con process running
2021/12/04 10:06:27 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/04 10:06:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:06:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:06:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:06:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:06:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:06:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:06:50 Waiting for dst bucket item count to get to: 0 curr count: 19650
2021/12/04 10:06:55 Waiting for dst bucket item count to get to: 0 curr count: 19650
2021/12/04 10:06:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (169.79s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/04 10:07:05 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 10:07:05 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/04 10:07:05 No eventing-con process running
2021/12/04 10:07:05 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/04 10:07:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:07:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:07:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:07:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:07:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:07:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:07:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:07:24 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:07:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 10:07:30 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 10:07:35 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 10:07:40 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 10:07:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 10:07:45 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/04 10:07:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:07:45 Starting bucket ops to source bucket
2021/12/04 10:07:45 Removing node(s): [] from the cluster
2021/12/04 10:07:50 Rebalance progress: 12.5
2021/12/04 10:07:55 Rebalance progress: 12.5
2021/12/04 10:08:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:08:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:08:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:08:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:08:06 Rebalance progress: 12.5
2021/12/04 10:08:11 Rebalance progress: 12.5
2021/12/04 10:08:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:08:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:08:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:08:16 Waiting for src bucket item count to get to: 50000 curr count: 30325
2021/12/04 10:08:21 Waiting for src bucket item count to get to: 50000 curr count: 41609
2021/12/04 10:08:26 Waiting for src bucket item count to get to: 50000 curr count: 41609
2021/12/04 10:08:36 Waiting for dst bucket item count to get to: 50000 curr count: 30567
2021/12/04 10:08:41 Waiting for dst bucket item count to get to: 50000 curr count: 30567
2021/12/04 10:08:46 Waiting for dst bucket item count to get to: 50000 curr count: 37423
2021/12/04 10:08:51 Waiting for dst bucket item count to get to: 50000 curr count: 37423
2021/12/04 10:08:51 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 10:08:51 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 10:08:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:08:56 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 10:09:01 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 10:09:06 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 10:09:11 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 10:09:16 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 10:09:21 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 10:09:21 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/04 10:09:21 No eventing-con process running
2021/12/04 10:09:21 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/04 10:09:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:09:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:09:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:09:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:09:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:09:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:09:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:09:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (169.25s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/04 10:09:50 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:09:50 Removing node(s): [] from the cluster
2021/12/04 10:09:55 Rebalance progress: 100
2021/12/04 10:10:00 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 10:10:00 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/04 10:10:00 No eventing-con process running
2021/12/04 10:10:00 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/04 10:10:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:10:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:10:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:10:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:10:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:10:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:10:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:10:19 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:10:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 10:10:25 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 10:10:30 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 10:10:35 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 10:10:40 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 10:10:40 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/04 10:10:40 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:10:40 Starting bucket ops to source bucket
2021/12/04 10:10:45 Rebalance progress: 12.5
2021/12/04 10:10:50 Rebalance progress: 12.5
2021/12/04 10:10:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:10:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:10:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:10:55 Waiting for src bucket item count to get to: 50000 curr count: 12164
2021/12/04 10:11:00 Waiting for src bucket item count to get to: 50000 curr count: 12164
2021/12/04 10:11:05 Waiting for src bucket item count to get to: 50000 curr count: 25641
2021/12/04 10:11:10 Waiting for src bucket item count to get to: 50000 curr count: 36623
2021/12/04 10:11:15 Waiting for src bucket item count to get to: 50000 curr count: 36623
2021/12/04 10:11:20 Waiting for src bucket item count to get to: 50000 curr count: 47083
2021/12/04 10:11:25 Waiting for src bucket item count to get to: 50000 curr count: 47083
2021/12/04 10:11:30 Waiting for src bucket item count to get to: 50000 curr count: 47083
2021/12/04 10:11:40 Waiting for dst bucket item count to get to: 50000 curr count: 36789
2021/12/04 10:11:45 Waiting for dst bucket item count to get to: 50000 curr count: 36789
2021/12/04 10:11:50 Waiting for dst bucket item count to get to: 50000 curr count: 45891
2021/12/04 10:11:55 Waiting for dst bucket item count to get to: 50000 curr count: 45891
2021/12/04 10:11:55 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 10:11:55 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 10:11:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:12:00 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 10:12:05 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 10:12:10 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 10:12:15 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 10:12:20 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 10:12:25 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 10:12:25 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/04 10:12:25 No eventing-con process running
2021/12/04 10:12:25 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/04 10:12:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:12:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:12:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:12:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:12:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:12:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:12:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:12:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (184.34s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/04 10:12:59 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 10:12:59 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/04 10:12:59 No eventing-con process running
2021/12/04 10:12:59 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/04 10:13:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:13:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:13:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:13:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:13:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:13:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:13:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:13:18 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:13:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 10:13:24 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 10:13:29 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 10:13:34 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 10:13:39 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 10:13:39 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/04 10:13:39 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:13:39 Starting bucket ops to source bucket
2021/12/04 10:13:39 Removing node(s): [] from the cluster
2021/12/04 10:13:44 Rebalance progress: 12.5
2021/12/04 10:13:49 Rebalance progress: 12.5
2021/12/04 10:13:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:13:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:13:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:13:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 10:13:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:14:00 Rebalance progress: 12.5
2021/12/04 10:14:05 Rebalance progress: 12.5
2021/12/04 10:14:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:14:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:14:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:14:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 10:14:15 Rebalance progress: 12.5
2021/12/04 10:14:20 Rebalance progress: 12.5
2021/12/04 10:14:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:14:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:14:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:14:25 Waiting for src bucket item count to get to: 50000 curr count: 48007
2021/12/04 10:14:30 Waiting for src bucket item count to get to: 50000 curr count: 48007
2021/12/04 10:14:40 Waiting for dst bucket item count to get to: 50000 curr count: 34729
2021/12/04 10:14:45 Waiting for dst bucket item count to get to: 50000 curr count: 34729
2021/12/04 10:14:50 Waiting for dst bucket item count to get to: 50000 curr count: 42105
2021/12/04 10:14:55 Waiting for dst bucket item count to get to: 50000 curr count: 42105
2021/12/04 10:14:55 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 10:14:55 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 10:14:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:15:00 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 10:15:05 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 10:15:10 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 10:15:15 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 10:15:20 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 10:15:25 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 10:15:25 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/04 10:15:25 No eventing-con process running
2021/12/04 10:15:26 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/04 10:15:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:15:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:15:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:15:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:15:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:15:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:15:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:15:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (180.18s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/04 10:15:59 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 10:15:59 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/04 10:15:59 No eventing-con process running
2021/12/04 10:15:59 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/04 10:16:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:16:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:16:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:16:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:16:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:16:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:16:18 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:16:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 10:16:24 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 10:16:29 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 10:16:34 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 10:16:39 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 10:16:39 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/04 10:16:39 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 10:16:39 Starting bucket ops to source bucket
2021/12/04 10:16:40 Removing node(s): [] from the cluster
2021/12/04 10:16:45 Rebalance progress: 12.5
2021/12/04 10:16:50 Rebalance progress: 12.5
2021/12/04 10:16:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:16:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:16:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:16:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:17:00 Rebalance progress: 12.5
2021/12/04 10:17:05 Rebalance progress: 12.5
2021/12/04 10:17:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:17:10 Waiting for src bucket item count to get to: 50000 curr count: 26143
2021/12/04 10:17:15 Waiting for src bucket item count to get to: 50000 curr count: 35339
2021/12/04 10:17:20 Waiting for src bucket item count to get to: 50000 curr count: 35339
2021/12/04 10:17:25 Waiting for src bucket item count to get to: 50000 curr count: 47941
2021/12/04 10:17:30 Waiting for src bucket item count to get to: 50000 curr count: 47941
2021/12/04 10:17:40 Waiting for dst bucket item count to get to: 50000 curr count: 42215
2021/12/04 10:17:45 Waiting for dst bucket item count to get to: 50000 curr count: 42215
2021/12/04 10:17:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 10:17:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 10:17:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:17:50 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 10:17:55 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 10:18:00 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 10:18:05 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 10:18:10 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 10:18:15 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 10:18:15 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/04 10:18:15 No eventing-con process running
2021/12/04 10:18:15 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/04 10:18:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:18:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:18:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:18:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:18:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:18:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:18:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:18:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.59s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/04 10:18:44 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 10:18:44 Removing node(s): [] from the cluster
2021/12/04 10:18:49 Rebalance progress: 100
2021/12/04 10:18:54 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 10:18:54 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/04 10:18:54 No eventing-con process running
2021/12/04 10:18:54 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/04 10:18:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:19:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:19:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:19:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:19:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:19:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:19:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:19:13 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:19:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 10:19:19 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 10:19:24 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 10:19:29 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 10:19:34 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 10:19:34 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/04 10:19:34 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:19:34 Starting bucket ops to source bucket
2021/12/04 10:19:39 Rebalance progress: 12.5
2021/12/04 10:19:44 Rebalance progress: 12.5
2021/12/04 10:19:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:19:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:19:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:19:49 Waiting for src bucket item count to get to: 50000 curr count: 6098
2021/12/04 10:19:54 Waiting for src bucket item count to get to: 50000 curr count: 19270
2021/12/04 10:19:59 Waiting for src bucket item count to get to: 50000 curr count: 19270
2021/12/04 10:20:04 Waiting for src bucket item count to get to: 50000 curr count: 30867
2021/12/04 10:20:09 Waiting for src bucket item count to get to: 50000 curr count: 30867
2021/12/04 10:20:14 Waiting for src bucket item count to get to: 50000 curr count: 42328
2021/12/04 10:20:19 Waiting for src bucket item count to get to: 50000 curr count: 42328
2021/12/04 10:20:24 Waiting for src bucket item count to get to: 50000 curr count: 42328
2021/12/04 10:20:34 Waiting for dst bucket item count to get to: 50000 curr count: 34421
2021/12/04 10:20:39 Waiting for dst bucket item count to get to: 50000 curr count: 34421
2021/12/04 10:20:44 Waiting for dst bucket item count to get to: 50000 curr count: 44290
2021/12/04 10:20:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 10:20:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 10:20:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:20:49 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 10:20:54 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 10:21:00 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 10:21:05 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 10:21:10 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 10:21:15 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 10:21:15 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/04 10:21:15 No eventing-con process running
2021/12/04 10:21:15 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/04 10:21:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:21:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:21:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:21:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:21:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:21:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:21:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:21:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.60s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/04 10:21:48 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 10:21:48 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/04 10:21:48 No eventing-con process running
2021/12/04 10:21:48 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/04 10:21:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:21:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:21:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:21:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:22:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:22:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:22:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:22:07 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:22:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 10:22:13 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 10:22:18 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 10:22:23 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 10:22:28 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 10:22:28 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/04 10:22:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 10:22:28 Starting bucket ops to source bucket
2021/12/04 10:22:29 Removing node(s): [] from the cluster
2021/12/04 10:22:34 Rebalance progress: 12.5
2021/12/04 10:22:39 Rebalance progress: 12.5
2021/12/04 10:22:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:22:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:22:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:22:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 10:22:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:22:49 Rebalance progress: 12.5
2021/12/04 10:22:54 Rebalance progress: 12.5
2021/12/04 10:22:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:22:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:22:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:22:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 10:23:04 Rebalance progress: 12.5
2021/12/04 10:23:09 Rebalance progress: 12.5
2021/12/04 10:23:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:23:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:23:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:23:14 Waiting for src bucket item count to get to: 50000 curr count: 47659
2021/12/04 10:23:19 Waiting for src bucket item count to get to: 50000 curr count: 47659
2021/12/04 10:23:29 Waiting for dst bucket item count to get to: 50000 curr count: 43484
2021/12/04 10:23:34 Waiting for dst bucket item count to get to: 50000 curr count: 43484
2021/12/04 10:23:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 10:23:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 10:23:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:23:40 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 10:23:45 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 10:23:50 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 10:23:55 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 10:24:00 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 10:24:05 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 10:24:05 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/04 10:24:05 No eventing-con process running
2021/12/04 10:24:05 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/04 10:24:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:24:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:24:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:24:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:24:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:24:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:24:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (170.16s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/04 10:24:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 10:24:39 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/04 10:24:39 No eventing-con process running
2021/12/04 10:24:39 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/04 10:24:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:24:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:24:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:24:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:24:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:24:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:24:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:24:57 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:24:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 10:25:03 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 10:25:08 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 10:25:13 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 10:25:18 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 10:25:18 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/04 10:25:18 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:25:18 Starting bucket ops to source bucket
2021/12/04 10:25:19 Removing node(s): [] from the cluster
2021/12/04 10:25:24 Rebalance progress: 12.5
2021/12/04 10:25:29 Rebalance progress: 12.5
2021/12/04 10:25:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:25:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:25:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:25:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:25:39 Rebalance progress: 12.5
2021/12/04 10:25:44 Rebalance progress: 12.5
2021/12/04 10:25:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:25:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:25:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:25:49 Waiting for src bucket item count to get to: 50000 curr count: 26163
2021/12/04 10:25:54 Waiting for src bucket item count to get to: 50000 curr count: 35248
2021/12/04 10:25:59 Waiting for src bucket item count to get to: 50000 curr count: 35248
2021/12/04 10:26:04 Waiting for src bucket item count to get to: 50000 curr count: 47713
2021/12/04 10:26:09 Waiting for src bucket item count to get to: 50000 curr count: 47713
2021/12/04 10:26:19 Waiting for dst bucket item count to get to: 50000 curr count: 40009
2021/12/04 10:26:24 Waiting for dst bucket item count to get to: 50000 curr count: 40009
2021/12/04 10:26:29 Waiting for dst bucket item count to get to: 50000 curr count: 48138
2021/12/04 10:26:34 Waiting for dst bucket item count to get to: 50000 curr count: 48138
2021/12/04 10:26:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 10:26:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 10:26:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:26:39 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 10:26:44 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 10:26:50 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 10:26:55 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 10:27:00 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 10:27:05 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 10:27:05 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/04 10:27:05 No eventing-con process running
2021/12/04 10:27:05 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/04 10:27:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:27:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:27:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:27:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:27:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:27:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:27:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:27:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (179.81s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/04 10:27:33 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:27:34 Removing node(s): [] from the cluster
2021/12/04 10:27:39 Rebalance progress: 100
2021/12/04 10:27:44 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 10:27:44 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/04 10:27:44 No eventing-con process running
2021/12/04 10:27:44 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/04 10:27:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:27:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:27:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:27:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:27:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:27:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:28:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:28:02 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:28:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 10:28:08 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 10:28:13 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 10:28:18 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 10:28:23 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 10:28:23 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/04 10:28:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:28:23 Starting bucket ops to source bucket
2021/12/04 10:28:29 Rebalance progress: 12.5
2021/12/04 10:28:34 Rebalance progress: 12.5
2021/12/04 10:28:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:28:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:28:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:28:39 Waiting for src bucket item count to get to: 50000 curr count: 7474
2021/12/04 10:28:44 Waiting for src bucket item count to get to: 50000 curr count: 19758
2021/12/04 10:28:49 Waiting for src bucket item count to get to: 50000 curr count: 19758
2021/12/04 10:28:54 Waiting for src bucket item count to get to: 50000 curr count: 31030
2021/12/04 10:28:59 Waiting for src bucket item count to get to: 50000 curr count: 31030
2021/12/04 10:29:04 Waiting for src bucket item count to get to: 50000 curr count: 31030
2021/12/04 10:29:09 Waiting for src bucket item count to get to: 50000 curr count: 42203
2021/12/04 10:29:14 Waiting for src bucket item count to get to: 50000 curr count: 42203
2021/12/04 10:29:24 Waiting for dst bucket item count to get to: 50000 curr count: 39302
2021/12/04 10:29:29 Waiting for dst bucket item count to get to: 50000 curr count: 39302
2021/12/04 10:29:34 Waiting for dst bucket item count to get to: 50000 curr count: 49244
2021/12/04 10:29:39 Waiting for dst bucket item count to get to: 50000 curr count: 49244
2021/12/04 10:29:39 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 10:29:39 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 10:29:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:29:44 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 10:29:49 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 10:29:54 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 10:29:59 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 10:30:04 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 10:30:04 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/04 10:30:04 No eventing-con process running
2021/12/04 10:30:04 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/04 10:30:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:30:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:30:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:30:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:30:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:30:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:30:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:30:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.45s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/04 10:30:38 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 10:30:38 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/04 10:30:38 No eventing-con process running
2021/12/04 10:30:38 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/04 10:30:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:30:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:30:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:30:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:30:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:30:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:30:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:30:56 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:30:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 10:31:03 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 10:31:08 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 10:31:13 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 10:31:18 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 10:31:18 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/04 10:31:18 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:31:18 Starting bucket ops to source bucket
2021/12/04 10:31:18 Removing node(s): [] from the cluster
2021/12/04 10:31:23 Rebalance progress: 12.5
2021/12/04 10:31:28 Rebalance progress: 12.5
2021/12/04 10:31:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:31:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:31:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:31:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 10:31:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:31:39 Rebalance progress: 12.5
2021/12/04 10:31:44 Rebalance progress: 12.5
2021/12/04 10:31:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:31:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:31:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:31:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 10:31:54 Rebalance progress: 12.5
2021/12/04 10:31:59 Rebalance progress: 12.5
2021/12/04 10:32:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:32:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:32:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:32:04 Waiting for src bucket item count to get to: 50000 curr count: 47273
2021/12/04 10:32:09 Waiting for src bucket item count to get to: 50000 curr count: 47273
2021/12/04 10:32:19 Waiting for dst bucket item count to get to: 50000 curr count: 46373
2021/12/04 10:32:24 Waiting for dst bucket item count to get to: 50000 curr count: 46373
2021/12/04 10:32:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 10:32:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 10:32:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:32:30 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 10:32:35 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 10:32:40 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 10:32:45 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 10:32:50 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 10:32:50 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/04 10:32:50 No eventing-con process running
2021/12/04 10:32:50 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/04 10:32:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:32:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:33:02 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:33:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:33:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:33:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:33:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:33:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 10:33:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (170.68s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/04 10:33:28 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:33:28 App: bucket_op_on_update got un-deployed
2021/12/04 10:33:28 No eventing-con process running
2021/12/04 10:33:28 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 10:33:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:33:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:33:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:33:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:33:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:33:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:33:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:33:47 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:33:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 10:33:53 Starting bucket ops to source bucket
2021/12/04 10:33:58 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:34:03 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:34:08 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:34:08 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:34:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:34:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:34:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:34:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:34:14 Rebalance progress: 12.5
2021/12/04 10:34:19 Rebalance progress: 12.5
2021/12/04 10:34:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:34:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:34:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:34:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 10:34:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:34:29 Rebalance progress: 12.5
2021/12/04 10:34:34 Rebalance progress: 12.5
2021/12/04 10:34:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:34:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:34:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:34:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 10:34:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:34:44 Rebalance progress: 12.5
2021/12/04 10:34:49 Rebalance progress: 12.5
2021/12/04 10:34:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:34:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:34:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:34:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:34:59 Rebalance progress: 12.5
2021/12/04 10:35:04 Rebalance progress: 12.5
2021/12/04 10:35:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:35:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:35:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:35:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 10:35:14 Rebalance progress: 12.5
2021/12/04 10:35:19 Rebalance progress: 12.5
2021/12/04 10:35:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:35:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:35:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:35:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 10:35:30 Rebalance progress: 12.5
2021/12/04 10:35:35 Rebalance progress: 12.5
2021/12/04 10:35:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:35:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:35:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:35:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:35:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:35:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:35:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:36:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:36:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:36:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:36:10 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:36:10 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:36:10 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 10:36:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:36:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:36:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10626
2021/12/04 10:36:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10626
2021/12/04 10:36:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:36:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:36:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:36:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10626
2021/12/04 10:36:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.03s)
=== RUN   TestKVForceFailover
2021/12/04 10:36:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:36:53 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:36:53 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:36:53 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 10:36:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:37:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:37:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:37:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:37:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:37:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:37:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:37:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:37:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:37:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:37:12 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:37:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 10:37:14 Starting bucket ops to source bucket
2021/12/04 10:37:19 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:37:24 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:37:29 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:37:34 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:37:34 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:37:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:37:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:37:34 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/04 10:37:54 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/04 10:37:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 10:37:59 Rebalance progress: 12.5
2021/12/04 10:38:04 Rebalance progress: 12.5
2021/12/04 10:38:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:38:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:38:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:38:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/04 10:38:14 Rebalance progress: 12.5
2021/12/04 10:38:19 Rebalance progress: 12.5
2021/12/04 10:38:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:38:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:38:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:38:24 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:38:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:38:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:38:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:38:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:38:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:38:50 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:38:50 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:38:50 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 10:38:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:38:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:39:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7043
2021/12/04 10:39:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7043
2021/12/04 10:39:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:39:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:39:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:39:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7043
2021/12/04 10:39:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.71s)
=== RUN   TestKVHardFailoverRecovery
2021/12/04 10:39:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:39:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:39:28 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:39:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 10:39:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:39:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:39:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:39:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:39:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:39:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:39:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:39:47 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:39:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 10:39:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 10:39:58 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:40:03 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:40:08 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:40:08 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:40:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:40:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:40:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:40:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:40:14 Rebalance progress: 12.5
2021/12/04 10:40:19 Rebalance progress: 12.5
2021/12/04 10:40:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:40:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:40:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:40:24 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/04 10:41:24 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/04 10:41:29 Rebalance progress: 12.5
2021/12/04 10:41:34 Rebalance progress: 12.5
2021/12/04 10:41:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:41:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:41:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:41:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:41:44 Rebalance progress: 12.5
2021/12/04 10:41:49 Rebalance progress: 12.5
2021/12/04 10:41:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:41:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:41:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:41:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:41:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:42:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:42:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:42:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:42:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:42:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:42:24 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:42:24 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:42:24 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 10:42:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:42:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:42:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12076
2021/12/04 10:42:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:42:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:42:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:42:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12076
2021/12/04 10:42:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.60s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/04 10:42:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:42:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:42:58 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:42:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 10:43:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:43:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:43:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:43:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:43:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:43:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:43:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:43:16 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:43:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 10:43:23 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 10:43:28 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:43:33 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:43:38 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:43:38 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:43:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:43:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:43:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:43:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 10:43:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 10:43:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:43:43 Rebalance progress: 12.5
2021/12/04 10:43:48 Rebalance progress: 12.5
2021/12/04 10:43:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:43:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:43:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:43:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/04 10:43:58 Rebalance progress: 12.5
2021/12/04 10:44:03 Rebalance progress: 12.5
2021/12/04 10:44:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:44:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:44:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:44:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:44:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:44:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:44:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:44:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:44:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:44:34 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:44:34 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:44:34 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 10:44:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:44:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:44:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4575
2021/12/04 10:44:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:44:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:44:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:44:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4575
2021/12/04 10:44:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (129.75s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/04 10:45:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:45:08 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:45:08 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:45:08 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 10:45:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:45:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:45:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:45:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:45:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:45:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:45:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:45:21 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:45:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 10:45:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 10:45:32 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:45:37 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:45:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:45:42 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:45:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:45:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:45:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:45:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:45:48 Rebalance progress: 12.5
2021/12/04 10:45:53 Rebalance progress: 12.5
2021/12/04 10:45:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:45:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:45:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:45:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 10:45:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:46:03 Rebalance progress: 12.5
2021/12/04 10:46:08 Rebalance progress: 12.5
2021/12/04 10:46:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:46:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:46:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:46:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 10:46:18 Rebalance progress: 12.5
2021/12/04 10:46:23 Rebalance progress: 12.5
2021/12/04 10:46:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:46:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:46:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:46:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:46:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:46:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:46:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:46:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:46:59 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:46:59 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:46:59 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 10:47:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:47:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:47:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3984
2021/12/04 10:47:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3984
2021/12/04 10:47:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:47:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:47:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:47:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6108
2021/12/04 10:47:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:47:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:47:32 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:47:32 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:47:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 10:47:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:47:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:47:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:47:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:47:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:47:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.56s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/04 10:47:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:47:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:47:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:47:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 10:47:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:48:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:48:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:48:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:48:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:48:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:48:15 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:48:15 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 10:48:15 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:48:15 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/04 10:48:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:48:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:48:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:48:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:48:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:48:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:48:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:48:28 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:48:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:48:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:48:29 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:48:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/04 10:48:36 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 10:48:41 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:48:46 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:48:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:48:51 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/04 10:48:56 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 10:49:01 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 10:49:06 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 10:49:06 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:49:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:49:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:49:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:49:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:49:11 Rebalance progress: 12.5
2021/12/04 10:49:16 Rebalance progress: 12.5
2021/12/04 10:49:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:49:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:49:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:49:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 10:49:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:49:26 Rebalance progress: 12.5
2021/12/04 10:49:31 Rebalance progress: 12.5
2021/12/04 10:49:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:49:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:49:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:49:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 10:49:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:49:42 Rebalance progress: 12.5
2021/12/04 10:49:47 Rebalance progress: 12.5
2021/12/04 10:49:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:49:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:49:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:49:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:49:57 Rebalance progress: 12.5
2021/12/04 10:50:02 Rebalance progress: 12.5
2021/12/04 10:50:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:50:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:50:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:50:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 10:50:12 Rebalance progress: 12.5
2021/12/04 10:50:17 Rebalance progress: 12.5
2021/12/04 10:50:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:50:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:50:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:50:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 10:50:27 Rebalance progress: 12.5
2021/12/04 10:50:32 Rebalance progress: 12.5
2021/12/04 10:50:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:50:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:50:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:50:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:50:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:50:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:50:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:50:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:51:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:51:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:51:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:51:07 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:51:08 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 10:51:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:51:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:51:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3700
2021/12/04 10:51:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3700
2021/12/04 10:51:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:51:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:51:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:51:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12148
2021/12/04 10:51:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:51:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:51:43 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:51:48 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:51:53 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:51:58 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:52:03 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:52:08 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:52:08 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 10:52:08 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:52:08 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/04 10:52:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:52:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:52:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:52:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:52:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:52:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (275.55s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/04 10:52:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:52:32 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:52:32 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:52:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 10:52:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:52:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:52:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:52:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:52:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:52:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:52:50 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:52:50 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 10:52:50 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:52:50 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/04 10:52:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:52:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:52:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:52:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:53:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:53:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:53:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:53:04 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:53:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:53:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:53:05 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:53:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/04 10:53:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 10:53:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:53:21 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:53:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:53:26 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/04 10:53:31 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 10:53:36 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 10:53:41 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 10:53:41 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:53:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:53:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:53:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:53:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:53:47 Rebalance progress: 12.5
2021/12/04 10:53:52 Rebalance progress: 12.5
2021/12/04 10:53:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:53:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:53:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:53:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 10:53:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:54:03 Rebalance progress: 12.5
2021/12/04 10:54:08 Rebalance progress: 12.5
2021/12/04 10:54:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:54:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:54:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:54:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 10:54:18 Rebalance progress: 12.5
2021/12/04 10:54:23 Rebalance progress: 12.5
2021/12/04 10:54:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:54:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:54:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:54:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:54:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:54:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:54:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:54:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:54:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:54:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:54:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:54:58 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:54:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 10:55:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:55:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:55:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5102
2021/12/04 10:55:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5102
2021/12/04 10:55:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:55:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:55:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:55:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7667
2021/12/04 10:55:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:55:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:55:34 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:55:39 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:55:44 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:55:49 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:55:54 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 10:55:54 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 10:55:54 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:55:54 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/04 10:55:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:56:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:56:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:56:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:56:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:56:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (225.95s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/04 10:56:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:56:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:56:18 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:56:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 10:56:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:56:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:56:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:56:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:56:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:56:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:56:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:56:36 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:56:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 10:56:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 10:56:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:56:52 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:56:57 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:56:57 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:56:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:56:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:56:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:56:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:57:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:57:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:57:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:57:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:57:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:57:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 10:57:33 Rebalance progress: 12.5
2021/12/04 10:57:38 Rebalance progress: 12.5
2021/12/04 10:57:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:57:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:57:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:57:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 10:57:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 10:57:49 Rebalance progress: 12.5
2021/12/04 10:57:54 Rebalance progress: 12.5
2021/12/04 10:57:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:57:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:57:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:57:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 10:58:04 Rebalance progress: 12.5
2021/12/04 10:58:09 Rebalance progress: 12.5
2021/12/04 10:58:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:58:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:58:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:58:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 10:58:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:58:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:58:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:58:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:58:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:58:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:58:44 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:58:44 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:58:44 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 10:58:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:58:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:58:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9127
2021/12/04 10:58:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:58:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:59:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:59:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9127
2021/12/04 10:59:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (180.10s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/04 10:59:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 10:59:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 10:59:18 eventingSetup.go:46: No eventing-con process running
2021/12/04 10:59:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 10:59:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:59:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:59:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 10:59:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:59:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:59:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:59:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 10:59:36 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 10:59:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 10:59:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 10:59:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:59:52 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:59:57 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 10:59:57 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:59:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 10:59:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 10:59:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 10:59:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 11:00:03 Rebalance progress: 12.5
2021/12/04 11:00:08 Rebalance progress: 12.5
2021/12/04 11:00:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:00:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:00:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 11:00:13 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:00:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:00:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 11:00:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 11:00:28 Rebalance progress: 12.5
2021/12/04 11:00:33 Rebalance progress: 12.5
2021/12/04 11:00:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:00:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:00:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 11:00:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 11:00:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 11:00:43 Rebalance progress: 12.5
2021/12/04 11:00:48 Rebalance progress: 12.5
2021/12/04 11:00:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:00:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:00:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 11:00:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 11:00:59 Rebalance progress: 12.5
2021/12/04 11:01:04 Rebalance progress: 12.5
2021/12/04 11:01:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:01:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:01:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 11:01:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:01:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:01:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:01:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:01:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:01:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:01:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:01:39 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 11:01:39 eventingSetup.go:46: No eventing-con process running
2021/12/04 11:01:39 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 11:01:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:01:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 11:01:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7672
2021/12/04 11:01:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7672
2021/12/04 11:01:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:01:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:02:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 11:02:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8606
2021/12/04 11:02:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (180.02s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/04 11:02:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 11:02:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 11:02:18 Rebalance progress: 100
2021/12/04 11:02:18 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:02:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 11:02:48 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:02:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 11:02:54 Removing node(s): [] from the cluster
2021/12/04 11:02:54 Starting bucket ops to source bucket
2021/12/04 11:02:59 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/04 11:03:04 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 11:03:09 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 11:03:09 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:03:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:03:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 11:03:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:03:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:03:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:03:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:03:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:03:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:03:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 11:03:40 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 11:03:40 eventingSetup.go:46: No eventing-con process running
2021/12/04 11:03:40 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 11:03:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:03:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 11:03:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1506
2021/12/04 11:03:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1506
2021/12/04 11:03:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:03:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 11:04:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 11:04:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1506
2021/12/04 11:04:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.62s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12225.696s

Rebalance tests took 3 hours, 23 minutes and 47 seconds.

Testing Succeeded


Version: versions-04.12.2021-03.23.cfg
Build Log: make-04.12.2021-03.23.log
Server Log: logs-04.12.2021-03.23.tar.gz

Finished