Building

Started building at 2021/12/10 17:32:25
Calculating base
Updating mirror
Basing run on 7.1.0-1887 e8f4b60fb
Updating tree for run 10.12.2021-17.32
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at 888c51e, changes since last good build: 
 888c51e7 Merge remote-tracking branch 'couchbase/unstable' into HEAD
 b674cfaf MB-48897: Disable plasma decompress during swapin config
 bcd0e86b MB-48897: Add plasma compress after swapin config
kv_engine is at 77fe9b6, changes since last good build: none
ns_server is at e02bf21, changes since last good build: none
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 04efaa5, changes since last good build: 
 04efaa575 Adding tests for System Events for GSI
 6ce1b0c45 Adding 8 node template file for cbas
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at de9f5f4, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable e19cf0df] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable e1dd4b00] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 5e04cbff] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 411ab738] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable b1270326] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable eca74149] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable 885d6ccf] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable fefadf02] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable 2890c67] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/10 17:48:28
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/10 17:49:57 Started functional
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:49424->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:49424->127.0.0.1:9000: read: connection reset by peer
2021/12/10 17:50:06 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/10 17:50:12 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/10 17:50:12 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 17:50:17 Removing node(s): [] from the cluster
2021/12/10 17:50:22 Rebalance progress: 0
2021/12/10 17:50:27 Rebalance progress: 0.687210648148148
2021/12/10 17:50:32 Rebalance progress: 5.443431712962963
2021/12/10 17:50:37 Rebalance progress: 9.25925925925926
2021/12/10 17:50:42 Rebalance progress: 13.671875
2021/12/10 17:50:47 Rebalance progress: 18.51851851851852
2021/12/10 17:50:52 Rebalance progress: 21.71947337962963
2021/12/10 17:50:57 Rebalance progress: 26.3671875
2021/12/10 17:51:02 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/10 17:51:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:51:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:51:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:51:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:52:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:52:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:52:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:52:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:53:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:53:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:53:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:53:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/10 17:54:08 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:54:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/10 17:54:14 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 17:54:19 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 17:54:24 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 17:54:29 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/10 17:54:29 App: TestAdvancedGetOps got deployed
2021/12/10 17:54:29 Starting bucket ops to source bucket
2021/12/10 17:54:34 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 17:54:39 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 17:54:44 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 17:54:44 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 17:54:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 17:54:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 17:54:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:54:50 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 17:54:55 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 17:55:00 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 17:55:05 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/10 17:55:05 App: TestAdvancedGetOps got un-deployed
2021/12/10 17:55:05 No eventing-con process running
2021/12/10 17:55:05 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/10 17:55:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 17:55:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 17:55:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:55:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 17:55:28 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/10 17:55:33 Waiting for dst bucket item count to get to: 0 curr count: 24
2021/12/10 17:55:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (265.14s)
=== RUN   TestAdvancedInsertOps
2021/12/10 17:55:33 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:55:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 17:55:39 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 17:55:44 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 17:55:49 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 17:55:54 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 17:55:54 App: TestAdvancedInsertOps got deployed
2021/12/10 17:55:54 Starting bucket ops to source bucket
2021/12/10 17:56:00 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 17:56:05 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/10 17:56:05 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/10 17:56:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 17:56:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 17:56:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:56:10 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 17:56:15 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 17:56:20 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 17:56:25 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 17:56:25 App: TestAdvancedInsertOps got un-deployed
2021/12/10 17:56:25 No eventing-con process running
2021/12/10 17:56:25 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/10 17:56:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 17:56:37 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 17:56:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:56:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 17:56:49 Waiting for dst bucket item count to get to: 0 curr count: 1009
2021/12/10 17:56:54 Waiting for dst bucket item count to get to: 0 curr count: 497
2021/12/10 17:56:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:56:54 Testing insert operation on source bucket
2021/12/10 17:56:54 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:56:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 17:57:00 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 17:57:05 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 17:57:10 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 17:57:15 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/10 17:57:15 App: TestAdvancedInsertOps got deployed
2021/12/10 17:57:15 Starting bucket ops to source bucket
2021/12/10 17:57:21 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/10 17:57:21 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/10 17:57:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 17:57:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 17:57:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:57:26 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 17:57:31 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 17:57:36 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 17:57:41 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/10 17:57:41 App: TestAdvancedInsertOps got un-deployed
2021/12/10 17:57:41 No eventing-con process running
2021/12/10 17:57:41 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/10 17:57:43 Context: Bucket flush, response code: 200 dump: 
2021/12/10 17:57:53 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/10 17:57:58 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 17:57:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:58:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 17:58:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (151.65s)
=== RUN   TestAdvancedUpsertOps
2021/12/10 17:58:05 Starting bucket ops to source bucket
2021/12/10 17:58:06 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:58:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 17:58:12 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 17:58:17 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 17:58:22 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 17:58:27 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 17:58:27 App: TestAdvancedUpsertOps got deployed
2021/12/10 17:58:27 Starting bucket ops to source bucket
2021/12/10 17:58:33 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 17:58:38 Waiting for dst bucket item count to get to: 0 curr count: 251
2021/12/10 17:58:43 Waiting for dst bucket item count to get to: 0 curr count: 251
2021/12/10 17:58:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:58:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 17:58:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 17:58:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:58:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 17:58:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 17:58:58 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 17:59:03 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 17:59:03 App: TestAdvancedUpsertOps got un-deployed
2021/12/10 17:59:03 No eventing-con process running
2021/12/10 17:59:03 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/10 17:59:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 17:59:14 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 17:59:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:59:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 17:59:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:59:21 Testing upsert operation on source bucket
2021/12/10 17:59:21 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 17:59:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/10 17:59:27 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 17:59:33 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 17:59:38 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 17:59:43 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/10 17:59:43 App: TestAdvancedUpsertOps got deployed
2021/12/10 17:59:43 Starting bucket ops to source bucket
2021/12/10 17:59:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 17:59:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 17:59:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 17:59:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 17:59:49 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 17:59:54 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 17:59:59 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 18:00:04 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/10 18:00:04 App: TestAdvancedUpsertOps got un-deployed
2021/12/10 18:00:04 No eventing-con process running
2021/12/10 18:00:04 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/10 18:00:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:00:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:00:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:00:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (132.94s)
=== RUN   TestAdvancedReplaceOps
2021/12/10 18:00:18 Starting bucket ops to source bucket
2021/12/10 18:00:19 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:00:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/10 18:00:25 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 18:00:30 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 18:00:35 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 18:00:40 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 18:00:40 App: TestAdvancedReplaceOps got deployed
2021/12/10 18:00:40 Starting bucket ops to source bucket
2021/12/10 18:00:46 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 18:00:51 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 18:00:56 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/10 18:00:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:00:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:00:56 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:00:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:01:01 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 18:01:06 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 18:01:11 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 18:01:16 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 18:01:16 App: TestAdvancedReplaceOps got un-deployed
2021/12/10 18:01:16 No eventing-con process running
2021/12/10 18:01:16 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/10 18:01:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:01:28 Waiting for dst bucket item count to get to: 0 curr count: 549
2021/12/10 18:01:33 Waiting for dst bucket item count to get to: 0 curr count: 37
2021/12/10 18:01:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:01:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:01:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:01:39 Testing upsert operation on source bucket
2021/12/10 18:01:39 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:01:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/10 18:01:46 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 18:01:51 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 18:01:56 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 18:02:01 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/10 18:02:01 App: TestAdvancedReplaceOps got deployed
2021/12/10 18:02:01 Starting bucket ops to source bucket
2021/12/10 18:02:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:02:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:02:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:02:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:02:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 18:02:12 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 18:02:17 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 18:02:22 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/10 18:02:22 App: TestAdvancedReplaceOps got un-deployed
2021/12/10 18:02:22 No eventing-con process running
2021/12/10 18:02:22 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/10 18:02:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:02:34 Waiting for dst bucket item count to get to: 0 curr count: 161
2021/12/10 18:02:39 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/10 18:02:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:02:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:02:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (147.89s)
=== RUN   TestAdvancedDeleteOps
2021/12/10 18:02:46 Starting bucket ops to source bucket
2021/12/10 18:02:46 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:02:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/10 18:02:52 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 18:02:57 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 18:03:02 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 18:03:07 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 18:03:07 App: TestAdvancedDeleteOps got deployed
2021/12/10 18:03:07 Starting bucket ops to source bucket
2021/12/10 18:03:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:03:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:03:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:03:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:03:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:03:17 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 18:03:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 18:03:27 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 18:03:32 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 18:03:32 App: TestAdvancedDeleteOps got un-deployed
2021/12/10 18:03:32 No eventing-con process running
2021/12/10 18:03:32 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/10 18:03:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:03:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:03:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:03:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:03:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:03:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:03:56 Testing delete operation on source bucket
2021/12/10 18:03:56 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:03:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/10 18:04:02 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 18:04:07 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 18:04:12 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 18:04:17 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/10 18:04:17 App: TestAdvancedDeleteOps got deployed
2021/12/10 18:04:17 Starting bucket ops to source bucket
2021/12/10 18:04:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:04:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:04:17 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:04:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:04:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 18:04:27 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 18:04:32 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 18:04:37 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/10 18:04:37 App: TestAdvancedDeleteOps got un-deployed
2021/12/10 18:04:37 No eventing-con process running
2021/12/10 18:04:37 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/10 18:04:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:04:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:04:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:04:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.56s)
=== RUN   TestEnoentAdvancedGet
2021/12/10 18:04:51 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:04:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/10 18:04:57 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 18:05:02 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 18:05:07 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 18:05:12 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 18:05:12 App: TestEnoentAdvancedGet got deployed
2021/12/10 18:05:12 Starting bucket ops to source bucket
2021/12/10 18:05:17 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:05:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:05:23 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 18:05:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:05:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:05:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:05:28 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 18:05:33 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 18:05:38 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 18:05:43 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 18:05:43 App: TestEnoentAdvancedGet got un-deployed
2021/12/10 18:05:43 No eventing-con process running
2021/12/10 18:05:43 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/10 18:05:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:05:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:05:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:05:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:06:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:06:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:06:06 Testing get enoent operation on source bucket
2021/12/10 18:06:06 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:06:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/10 18:06:12 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 18:06:17 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 18:06:22 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 18:06:27 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/10 18:06:27 App: TestEnoentAdvancedGet got deployed
2021/12/10 18:06:27 Starting bucket ops to source bucket
2021/12/10 18:06:33 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/10 18:06:38 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 18:06:43 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 18:06:43 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/10 18:06:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:06:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:06:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:06:48 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 18:06:53 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 18:06:58 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 18:07:03 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/10 18:07:03 App: TestEnoentAdvancedGet got un-deployed
2021/12/10 18:07:03 No eventing-con process running
2021/12/10 18:07:03 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/10 18:07:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:07:14 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 18:07:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:07:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:07:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (149.97s)
=== RUN   TestEnoentAdvancedReplace
2021/12/10 18:07:21 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:07:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 18:07:27 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 18:07:32 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 18:07:37 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 18:07:42 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 18:07:42 App: TestEnoentAdvancedReplace got deployed
2021/12/10 18:07:42 Starting bucket ops to source bucket
2021/12/10 18:07:47 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:07:52 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:07:52 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 18:07:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:07:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:07:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:07:58 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 18:08:03 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 18:08:08 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 18:08:13 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 18:08:13 App: TestEnoentAdvancedReplace got un-deployed
2021/12/10 18:08:13 No eventing-con process running
2021/12/10 18:08:13 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/10 18:08:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:08:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:08:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:08:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:08:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:08:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:08:36 Testing get enoent operation on source bucket
2021/12/10 18:08:36 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:08:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 18:08:42 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 18:08:47 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 18:08:52 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 18:08:57 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/10 18:08:57 App: TestEnoentAdvancedReplace got deployed
2021/12/10 18:08:57 Starting bucket ops to source bucket
2021/12/10 18:09:02 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/10 18:09:07 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 18:09:12 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/10 18:09:12 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/10 18:09:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:09:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:09:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:09:17 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 18:09:22 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 18:09:27 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 18:09:32 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/10 18:09:33 App: TestEnoentAdvancedReplace got un-deployed
2021/12/10 18:09:33 No eventing-con process running
2021/12/10 18:09:33 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/10 18:09:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:09:44 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 18:09:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:09:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:09:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:09:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (154.84s)
=== RUN   TestEnoentAdvancedDelete
2021/12/10 18:09:56 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:09:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 18:10:02 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 18:10:07 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 18:10:12 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 18:10:17 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 18:10:17 App: TestEnoentAdvancedDelete got deployed
2021/12/10 18:10:17 Starting bucket ops to source bucket
2021/12/10 18:10:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:10:27 Waiting for dst bucket item count to get to: 100 curr count: 62
2021/12/10 18:10:32 Waiting for dst bucket item count to get to: 100 curr count: 62
2021/12/10 18:10:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 18:10:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:10:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:10:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:10:37 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 18:10:42 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 18:10:47 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 18:10:52 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 18:10:52 App: TestEnoentAdvancedDelete got un-deployed
2021/12/10 18:10:52 No eventing-con process running
2021/12/10 18:10:52 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/10 18:10:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:11:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:11:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:11:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:11:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:11:21 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/10 18:11:26 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/10 18:11:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:11:26 Testing get enoent operation on source bucket
2021/12/10 18:11:26 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:11:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 18:11:32 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 18:11:37 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 18:11:42 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 18:11:47 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/10 18:11:47 App: TestEnoentAdvancedDelete got deployed
2021/12/10 18:11:47 Starting bucket ops to source bucket
2021/12/10 18:11:52 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/10 18:11:57 Waiting for dst bucket item count to get to: 200 curr count: 162
2021/12/10 18:12:02 Waiting for dst bucket item count to get to: 200 curr count: 162
2021/12/10 18:12:02 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/10 18:12:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:12:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:12:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:12:07 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 18:12:12 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 18:12:17 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 18:12:22 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/10 18:12:22 App: TestEnoentAdvancedDelete got un-deployed
2021/12/10 18:12:22 No eventing-con process running
2021/12/10 18:12:22 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/10 18:12:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:12:34 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 18:12:39 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:12:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:12:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:12:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (169.85s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/10 18:12:46 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:12:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/10 18:12:52 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 18:12:57 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 18:13:02 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 18:13:07 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 18:13:07 App: TestAdvancedInsertKeyExist got deployed
2021/12/10 18:13:07 Starting bucket ops to source bucket
2021/12/10 18:13:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:13:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:13:07 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:13:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:13:12 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 18:13:17 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 18:13:22 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 18:13:27 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 18:13:27 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/10 18:13:27 No eventing-con process running
2021/12/10 18:13:27 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/10 18:13:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:13:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:13:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:13:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:13:42 Pumping document in the destination bucket
2021/12/10 18:13:42 Starting bucket ops to source bucket
2021/12/10 18:13:42 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:13:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/10 18:13:48 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 18:13:53 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 18:13:58 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 18:14:03 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/10 18:14:03 App: TestAdvancedInsertKeyExist got deployed
2021/12/10 18:14:03 Starting bucket ops to source bucket
2021/12/10 18:14:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:14:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:14:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:14:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:14:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:14:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:14:18 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 18:14:23 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 18:14:28 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 18:14:33 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/10 18:14:33 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/10 18:14:33 No eventing-con process running
2021/12/10 18:14:33 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/10 18:14:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:14:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:14:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:14:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:14:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (125.84s)
=== RUN   TestExipryGet
2021/12/10 18:14:52 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:14:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/10 18:14:58 Waiting for app: TestExipryGet to get deployed
2021/12/10 18:15:03 Waiting for app: TestExipryGet to get deployed
2021/12/10 18:15:08 Waiting for app: TestExipryGet to get deployed
2021/12/10 18:15:13 Waiting for app: TestExipryGet to get deployed
2021/12/10 18:15:13 App: TestExipryGet got deployed
2021/12/10 18:15:13 Starting bucket ops to source bucket
2021/12/10 18:15:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:16:03 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:16:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:16:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:16:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:16:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:16:08 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 18:16:13 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 18:16:18 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 18:16:23 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 18:16:23 App: TestExipryGet got un-deployed
2021/12/10 18:16:23 No eventing-con process running
2021/12/10 18:16:23 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/10 18:16:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:16:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:16:32 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:16:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:16:37 Pumping document in the destination bucket for expiry get
2021/12/10 18:16:37 Starting bucket ops to source bucket
2021/12/10 18:16:37 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:16:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/10 18:16:44 Waiting for app: TestExipryGet to get deployed
2021/12/10 18:16:49 Waiting for app: TestExipryGet to get deployed
2021/12/10 18:16:54 Waiting for app: TestExipryGet to get deployed
2021/12/10 18:16:59 Waiting for app: TestExipryGet to get deployed
2021/12/10 18:16:59 App: TestExipryGet got deployed
2021/12/10 18:16:59 Starting bucket ops to source bucket
2021/12/10 18:17:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:17:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:17:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:17:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:17:49 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:17:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:17:54 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 18:17:59 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 18:18:04 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 18:18:09 Waiting for app: TestExipryGet to get un-deployed
2021/12/10 18:18:09 App: TestExipryGet got un-deployed
2021/12/10 18:18:09 No eventing-con process running
2021/12/10 18:18:09 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/10 18:18:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:18:21 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/10 18:18:26 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/10 18:18:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:18:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:18:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (220.69s)
=== RUN   TestExipryInsert
2021/12/10 18:18:32 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:18:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/10 18:18:39 Waiting for app: TestExipryInsert to get deployed
2021/12/10 18:18:44 Waiting for app: TestExipryInsert to get deployed
2021/12/10 18:18:49 Waiting for app: TestExipryInsert to get deployed
2021/12/10 18:18:54 Waiting for app: TestExipryInsert to get deployed
2021/12/10 18:18:54 App: TestExipryInsert got deployed
2021/12/10 18:18:54 Starting bucket ops to source bucket
2021/12/10 18:19:39 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/10 18:19:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:19:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:19:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:19:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:19:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:19:49 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 18:19:54 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 18:19:59 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 18:20:04 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 18:20:04 App: TestExipryInsert got un-deployed
2021/12/10 18:20:04 No eventing-con process running
2021/12/10 18:20:04 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/10 18:20:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:20:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:20:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:20:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:20:18 Testing on the destination bucket for expiry insert
2021/12/10 18:20:18 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:20:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/10 18:20:24 Waiting for app: TestExipryInsert to get deployed
2021/12/10 18:20:29 Waiting for app: TestExipryInsert to get deployed
2021/12/10 18:20:34 Waiting for app: TestExipryInsert to get deployed
2021/12/10 18:20:39 Waiting for app: TestExipryInsert to get deployed
2021/12/10 18:20:39 App: TestExipryInsert got deployed
2021/12/10 18:20:39 Starting bucket ops to source bucket
2021/12/10 18:21:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:21:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:21:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:21:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:21:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:21:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:21:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:21:40 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 18:21:45 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 18:21:50 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 18:21:55 Waiting for app: TestExipryInsert to get un-deployed
2021/12/10 18:21:55 App: TestExipryInsert got un-deployed
2021/12/10 18:21:55 No eventing-con process running
2021/12/10 18:21:55 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/10 18:21:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:22:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:22:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:22:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:22:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.67s)
=== RUN   TestExipryReplace
2021/12/10 18:22:13 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:22:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 18:22:19 Waiting for app: TestExipryReplace to get deployed
2021/12/10 18:22:24 Waiting for app: TestExipryReplace to get deployed
2021/12/10 18:22:29 Waiting for app: TestExipryReplace to get deployed
2021/12/10 18:22:34 Waiting for app: TestExipryReplace to get deployed
2021/12/10 18:22:34 App: TestExipryReplace got deployed
2021/12/10 18:22:34 Starting bucket ops to source bucket
2021/12/10 18:23:19 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:23:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:23:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:23:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:23:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:23:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:23:30 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 18:23:35 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 18:23:40 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 18:23:45 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 18:23:45 App: TestExipryReplace got un-deployed
2021/12/10 18:23:45 No eventing-con process running
2021/12/10 18:23:45 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/10 18:23:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:23:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:23:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:23:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:23:59 Testing on the destination bucket for expiry insert
2021/12/10 18:23:59 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:24:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 18:24:05 Waiting for app: TestExipryReplace to get deployed
2021/12/10 18:24:10 Waiting for app: TestExipryReplace to get deployed
2021/12/10 18:24:15 Waiting for app: TestExipryReplace to get deployed
2021/12/10 18:24:20 Waiting for app: TestExipryReplace to get deployed
2021/12/10 18:24:20 App: TestExipryReplace got deployed
2021/12/10 18:24:20 Starting bucket ops to source bucket
2021/12/10 18:25:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:25:10 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:25:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:25:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:25:10 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:25:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:25:15 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 18:25:20 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 18:25:25 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 18:25:30 Waiting for app: TestExipryReplace to get un-deployed
2021/12/10 18:25:30 App: TestExipryReplace got un-deployed
2021/12/10 18:25:30 No eventing-con process running
2021/12/10 18:25:30 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/10 18:25:32 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:25:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:25:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:25:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:25:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:25:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.46s)
=== RUN   TestExipryUpsert
2021/12/10 18:25:54 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:25:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/10 18:26:00 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 18:26:05 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 18:26:10 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 18:26:15 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 18:26:15 App: TestExipryUpsert got deployed
2021/12/10 18:26:15 Starting bucket ops to source bucket
2021/12/10 18:27:00 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:27:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:27:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:27:00 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:27:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:27:05 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 18:27:10 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 18:27:15 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 18:27:20 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 18:27:20 App: TestExipryUpsert got un-deployed
2021/12/10 18:27:20 No eventing-con process running
2021/12/10 18:27:20 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/10 18:27:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:27:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:27:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:27:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:27:34 Testing on the destination bucket for expiry insert
2021/12/10 18:27:34 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:27:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/10 18:27:40 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 18:27:45 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 18:27:50 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 18:27:55 Waiting for app: TestExipryUpsert to get deployed
2021/12/10 18:27:55 App: TestExipryUpsert got deployed
2021/12/10 18:27:55 Starting bucket ops to source bucket
2021/12/10 18:28:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:28:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:28:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:28:40 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:28:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:28:46 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 18:28:51 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 18:28:56 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 18:29:01 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/10 18:29:01 App: TestExipryUpsert got un-deployed
2021/12/10 18:29:01 No eventing-con process running
2021/12/10 18:29:01 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/10 18:29:02 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:29:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:29:17 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:29:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:29:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:29:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (210.57s)
=== RUN   TestCasReplace
2021/12/10 18:29:24 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:29:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 18:29:30 Waiting for app: TestCasReplace to get deployed
2021/12/10 18:29:35 Waiting for app: TestCasReplace to get deployed
2021/12/10 18:29:40 Waiting for app: TestCasReplace to get deployed
2021/12/10 18:29:45 Waiting for app: TestCasReplace to get deployed
2021/12/10 18:29:45 App: TestCasReplace got deployed
2021/12/10 18:29:45 Starting bucket ops to source bucket
2021/12/10 18:29:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:29:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:29:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:29:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:29:51 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 18:29:56 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 18:30:01 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 18:30:06 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 18:30:06 App: TestCasReplace got un-deployed
2021/12/10 18:30:06 No eventing-con process running
2021/12/10 18:30:06 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/10 18:30:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:30:17 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/10 18:30:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:30:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:30:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:30:24 Testing on the destination bucket for upsert with cas
2021/12/10 18:30:24 Starting bucket ops to source bucket
2021/12/10 18:30:24 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:30:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/10 18:30:30 Waiting for app: TestCasReplace to get deployed
2021/12/10 18:30:35 Waiting for app: TestCasReplace to get deployed
2021/12/10 18:30:40 Waiting for app: TestCasReplace to get deployed
2021/12/10 18:30:45 Waiting for app: TestCasReplace to get deployed
2021/12/10 18:30:45 App: TestCasReplace got deployed
2021/12/10 18:30:45 Starting bucket ops to source bucket
2021/12/10 18:30:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:30:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:30:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:30:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:30:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:30:56 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 18:31:01 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 18:31:06 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 18:31:11 Waiting for app: TestCasReplace to get un-deployed
2021/12/10 18:31:11 App: TestCasReplace got un-deployed
2021/12/10 18:31:11 No eventing-con process running
2021/12/10 18:31:11 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/10 18:31:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:31:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:31:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:31:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:31:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:31:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.16s)
=== RUN   TestCasDelete
2021/12/10 18:31:34 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:31:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 18:31:41 Waiting for app: TestCasDelete to get deployed
2021/12/10 18:31:46 Waiting for app: TestCasDelete to get deployed
2021/12/10 18:31:51 Waiting for app: TestCasDelete to get deployed
2021/12/10 18:31:56 Waiting for app: TestCasDelete to get deployed
2021/12/10 18:31:56 App: TestCasDelete got deployed
2021/12/10 18:31:56 Starting bucket ops to source bucket
2021/12/10 18:31:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:31:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:31:56 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:31:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:32:01 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 18:32:06 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 18:32:11 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 18:32:16 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 18:32:16 App: TestCasDelete got un-deployed
2021/12/10 18:32:16 No eventing-con process running
2021/12/10 18:32:16 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/10 18:32:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:32:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:32:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:32:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:32:30 Testing on the destination bucket for delete with cas
2021/12/10 18:32:30 Starting bucket ops to source bucket
2021/12/10 18:32:30 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:32:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 18:32:36 Waiting for app: TestCasDelete to get deployed
2021/12/10 18:32:41 Waiting for app: TestCasDelete to get deployed
2021/12/10 18:32:46 Waiting for app: TestCasDelete to get deployed
2021/12/10 18:32:51 Waiting for app: TestCasDelete to get deployed
2021/12/10 18:32:51 App: TestCasDelete got deployed
2021/12/10 18:32:51 Starting bucket ops to source bucket
2021/12/10 18:32:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:33:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:33:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:33:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:33:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 18:33:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:33:06 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 18:33:11 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 18:33:16 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 18:33:21 Waiting for app: TestCasDelete to get un-deployed
2021/12/10 18:33:21 App: TestCasDelete got un-deployed
2021/12/10 18:33:21 No eventing-con process running
2021/12/10 18:33:21 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/10 18:33:23 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:33:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:33:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:33:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:33:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:33:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.64s)
=== RUN   TestCountersIncrement
2021/12/10 18:33:45 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 18:33:51 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 18:33:51 Removing node(s): [] from the cluster
2021/12/10 18:33:56 Rebalance progress: 100
2021/12/10 18:33:56 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:33:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/10 18:34:03 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 18:34:08 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 18:34:13 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 18:34:18 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 18:34:18 App: TestCountersIncrement got deployed
2021/12/10 18:34:18 Starting bucket ops to source bucket
2021/12/10 18:34:24 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 18:34:29 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/10 18:34:34 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/10 18:34:34 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/10 18:34:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:34:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:34:39 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 18:34:44 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 18:34:49 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 18:34:54 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 18:34:54 App: TestCountersIncrement got un-deployed
2021/12/10 18:34:54 No eventing-con process running
2021/12/10 18:34:54 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/10 18:34:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:35:06 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/10 18:35:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:35:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:35:17 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/10 18:35:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:35:17 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:35:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/10 18:35:24 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 18:35:29 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 18:35:34 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 18:35:39 Waiting for app: TestCountersIncrement to get deployed
2021/12/10 18:35:39 App: TestCountersIncrement got deployed
2021/12/10 18:35:39 Starting bucket ops to source bucket
2021/12/10 18:35:45 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/10 18:35:50 Waiting for dst bucket item count to get to: 2048 curr count: 466
2021/12/10 18:35:55 Waiting for dst bucket item count to get to: 2048 curr count: 1490
2021/12/10 18:35:55 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/10 18:35:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:35:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:36:00 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 18:36:05 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 18:36:10 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 18:36:15 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/10 18:36:15 App: TestCountersIncrement got un-deployed
2021/12/10 18:36:15 No eventing-con process running
2021/12/10 18:36:15 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/10 18:36:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:36:27 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/10 18:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:36:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:36:39 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/10 18:36:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:36:39 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/10 18:36:44 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.78s)
=== RUN   TestCountersDecrement
2021/12/10 18:36:44 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 18:36:44 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 18:36:44 Removing node(s): [] from the cluster
2021/12/10 18:36:49 Rebalance progress: 100
2021/12/10 18:36:49 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:36:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/10 18:36:56 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 18:37:01 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 18:37:06 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 18:37:11 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 18:37:11 App: TestCountersDecrement got deployed
2021/12/10 18:37:11 Starting bucket ops to source bucket
2021/12/10 18:37:17 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 18:37:22 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/10 18:37:27 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/10 18:37:27 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/10 18:37:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:37:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:37:32 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 18:37:37 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 18:37:42 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 18:37:47 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 18:37:47 App: TestCountersDecrement got un-deployed
2021/12/10 18:37:47 No eventing-con process running
2021/12/10 18:37:47 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/10 18:37:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:37:59 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/10 18:38:04 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/10 18:38:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:38:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:38:16 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/10 18:38:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:38:16 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:38:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/10 18:38:22 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 18:38:27 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 18:38:32 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 18:38:37 Waiting for app: TestCountersDecrement to get deployed
2021/12/10 18:38:37 App: TestCountersDecrement got deployed
2021/12/10 18:38:37 Starting bucket ops to source bucket
2021/12/10 18:38:43 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/10 18:38:48 Waiting for dst bucket item count to get to: 2048 curr count: 1195
2021/12/10 18:38:53 Waiting for dst bucket item count to get to: 2048 curr count: 1195
2021/12/10 18:38:53 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/10 18:38:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:38:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:38:58 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 18:39:03 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 18:39:08 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 18:39:13 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/10 18:39:13 App: TestCountersDecrement got un-deployed
2021/12/10 18:39:13 No eventing-con process running
2021/12/10 18:39:13 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/10 18:39:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:39:25 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/10 18:39:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:39:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:39:37 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/10 18:39:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:39:37 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/10 18:39:42 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.19s)
=== RUN   TestCacheGet
2021/12/10 18:39:42 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:39:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/10 18:39:48 Waiting for app: TestCacheGet to get deployed
2021/12/10 18:39:53 Waiting for app: TestCacheGet to get deployed
2021/12/10 18:39:58 Waiting for app: TestCacheGet to get deployed
2021/12/10 18:40:03 Waiting for app: TestCacheGet to get deployed
2021/12/10 18:40:03 App: TestCacheGet got deployed
2021/12/10 18:40:03 Starting bucket ops to source bucket
2021/12/10 18:40:08 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/10 18:40:13 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/10 18:40:18 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/10 18:40:23 Waiting for dst bucket item count to get to: 20 curr count: 8
2021/12/10 18:40:28 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/10 18:40:33 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/10 18:40:33 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/10 18:40:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:40:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:40:38 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 18:40:43 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 18:40:48 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 18:40:54 Waiting for app: TestCacheGet to get un-deployed
2021/12/10 18:40:54 App: TestCacheGet got un-deployed
2021/12/10 18:40:54 No eventing-con process running
2021/12/10 18:40:54 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/10 18:40:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:41:05 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/10 18:41:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:41:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:41:17 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/10 18:41:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.20s)
=== RUN   TestCacheRYOW
2021/12/10 18:41:17 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:41:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/10 18:41:23 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 18:41:28 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 18:41:33 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 18:41:38 Waiting for app: TestCacheRYOW to get deployed
2021/12/10 18:41:38 App: TestCacheRYOW got deployed
2021/12/10 18:41:38 Starting bucket ops to source bucket
2021/12/10 18:41:43 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/10 18:41:49 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/10 18:41:49 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/10 18:41:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:41:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:41:54 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 18:41:59 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 18:42:04 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 18:42:09 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/10 18:42:09 App: TestCacheRYOW got un-deployed
2021/12/10 18:42:09 No eventing-con process running
2021/12/10 18:42:09 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/10 18:42:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:42:20 Waiting for dst bucket item count to get to: 0 curr count: 40
2021/12/10 18:42:25 Waiting for dst bucket item count to get to: 0 curr count: 40
2021/12/10 18:42:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:42:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:42:37 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:42:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.20s)
=== RUN   TestCacheExpiry
2021/12/10 18:42:37 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:42:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/10 18:42:43 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 18:42:48 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 18:42:53 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 18:42:58 Waiting for app: TestCacheExpiry to get deployed
2021/12/10 18:42:58 App: TestCacheExpiry got deployed
2021/12/10 18:42:58 Starting bucket ops to source bucket
2021/12/10 18:43:04 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:43:09 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/10 18:43:14 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/10 18:43:19 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/10 18:43:24 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/10 18:43:24 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 18:43:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:43:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:43:29 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 18:43:34 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 18:43:39 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 18:43:44 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/10 18:43:44 App: TestCacheExpiry got un-deployed
2021/12/10 18:43:44 No eventing-con process running
2021/12/10 18:43:44 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/10 18:43:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:43:56 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:43:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:43:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:44:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:44:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (90.11s)
=== RUN   TestCacheExpiryCustom
2021/12/10 18:44:07 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:44:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/10 18:44:14 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 18:44:19 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 18:44:24 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 18:44:29 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/10 18:44:29 App: TestCacheExpiryCustom got deployed
2021/12/10 18:44:29 Starting bucket ops to source bucket
2021/12/10 18:44:34 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:44:39 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/10 18:44:44 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/10 18:44:49 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/10 18:44:54 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/10 18:44:54 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 18:44:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:44:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:44:59 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 18:45:04 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 18:45:09 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 18:45:14 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/10 18:45:14 App: TestCacheExpiryCustom got un-deployed
2021/12/10 18:45:14 No eventing-con process running
2021/12/10 18:45:14 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/10 18:45:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:45:26 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:45:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:45:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:45:37 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:45:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (90.16s)
=== RUN   TestCacheOverflowCustom
2021/12/10 18:45:38 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:45:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/10 18:45:44 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 18:45:49 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 18:45:54 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 18:45:59 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/10 18:45:59 App: TestCacheOverflowCustom got deployed
2021/12/10 18:45:59 Starting bucket ops to source bucket
2021/12/10 18:46:04 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/10 18:46:09 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/10 18:46:14 Waiting for dst bucket item count to get to: 35 curr count: 9
2021/12/10 18:46:19 Waiting for dst bucket item count to get to: 35 curr count: 21
2021/12/10 18:46:24 Waiting for dst bucket item count to get to: 35 curr count: 29
2021/12/10 18:46:29 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/10 18:46:34 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/10 18:46:34 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/10 18:46:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:46:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:46:39 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 18:46:44 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 18:46:49 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 18:46:54 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/10 18:46:54 App: TestCacheOverflowCustom got un-deployed
2021/12/10 18:46:54 No eventing-con process running
2021/12/10 18:46:54 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/10 18:46:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:47:06 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/10 18:47:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:47:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:47:18 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/10 18:47:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.06s)
=== RUN   TestStrictMode
2021/12/10 18:47:18 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:47:19 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.07s)
=== RUN   TestDataTypes
2021/12/10 18:47:19 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:47:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/10 18:47:25 Waiting for app: TestDataTypes to get deployed
2021/12/10 18:47:30 Waiting for app: TestDataTypes to get deployed
2021/12/10 18:47:35 Waiting for app: TestDataTypes to get deployed
2021/12/10 18:47:40 Waiting for app: TestDataTypes to get deployed
2021/12/10 18:47:40 App: TestDataTypes got deployed
2021/12/10 18:47:40 Starting bucket ops to source bucket
2021/12/10 18:47:45 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/10 18:47:50 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/10 18:47:55 Waiting for dst bucket item count to get to: 18 curr count: 8
2021/12/10 18:47:55 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/10 18:47:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:47:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:48:00 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 18:48:05 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 18:48:10 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 18:48:15 Waiting for app: TestDataTypes to get un-deployed
2021/12/10 18:48:15 App: TestDataTypes got un-deployed
2021/12/10 18:48:15 No eventing-con process running
2021/12/10 18:48:15 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/10 18:48:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:48:27 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/10 18:48:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:48:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:48:39 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/10 18:48:44 Waiting for dst bucket item count to get to: 0 curr count: 7
2021/12/10 18:48:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (85.11s)
=== RUN   TestEnoentGet
2021/12/10 18:48:44 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:48:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/10 18:48:50 Waiting for app: TestEnoentGet to get deployed
2021/12/10 18:48:55 Waiting for app: TestEnoentGet to get deployed
2021/12/10 18:49:00 Waiting for app: TestEnoentGet to get deployed
2021/12/10 18:49:05 Waiting for app: TestEnoentGet to get deployed
2021/12/10 18:49:05 App: TestEnoentGet got deployed
2021/12/10 18:49:05 Starting bucket ops to source bucket
2021/12/10 18:49:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:49:15 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 18:49:15 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 18:49:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:49:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:49:20 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 18:49:25 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 18:49:30 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 18:49:35 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 18:49:35 App: TestEnoentGet got un-deployed
2021/12/10 18:49:35 No eventing-con process running
2021/12/10 18:49:35 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/10 18:49:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:49:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:49:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:49:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:49:59 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/10 18:50:04 Waiting for dst bucket item count to get to: 0 curr count: 46
2021/12/10 18:50:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:50:04 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:50:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/10 18:50:10 Waiting for app: TestEnoentGet to get deployed
2021/12/10 18:50:15 Waiting for app: TestEnoentGet to get deployed
2021/12/10 18:50:20 Waiting for app: TestEnoentGet to get deployed
2021/12/10 18:50:25 Waiting for app: TestEnoentGet to get deployed
2021/12/10 18:50:25 App: TestEnoentGet got deployed
2021/12/10 18:50:25 Starting bucket ops to source bucket
2021/12/10 18:50:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:50:35 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 18:50:35 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 18:50:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:50:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:50:40 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 18:50:45 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 18:50:50 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 18:50:55 Waiting for app: TestEnoentGet to get un-deployed
2021/12/10 18:50:55 App: TestEnoentGet got un-deployed
2021/12/10 18:50:55 No eventing-con process running
2021/12/10 18:50:55 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/10 18:50:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:51:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:51:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:51:09 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:51:19 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:51:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:51:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.18s)
=== RUN   TestEnoentDelete
2021/12/10 18:51:24 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:51:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 18:51:30 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 18:51:35 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 18:51:40 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 18:51:45 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 18:51:45 App: TestEnoentDelete got deployed
2021/12/10 18:51:45 Starting bucket ops to source bucket
2021/12/10 18:51:50 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:51:50 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 18:51:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:51:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:51:55 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 18:52:00 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 18:52:05 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 18:52:10 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 18:52:10 App: TestEnoentDelete got un-deployed
2021/12/10 18:52:11 No eventing-con process running
2021/12/10 18:52:11 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/10 18:52:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:52:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:52:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:52:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:52:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:52:39 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:52:44 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:52:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:52:44 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:52:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 18:52:50 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 18:52:55 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 18:53:00 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 18:53:05 Waiting for app: TestEnoentDelete to get deployed
2021/12/10 18:53:05 App: TestEnoentDelete got deployed
2021/12/10 18:53:05 Starting bucket ops to source bucket
2021/12/10 18:53:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:53:10 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 18:53:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:53:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:53:16 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 18:53:21 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 18:53:26 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 18:53:31 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/10 18:53:31 App: TestEnoentDelete got un-deployed
2021/12/10 18:53:31 No eventing-con process running
2021/12/10 18:53:31 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/10 18:53:32 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:53:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:53:47 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:53:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:53:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:53:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:54:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/10 18:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.28s)
=== RUN   TestError
2021/12/10 18:54:14 Waiting for app: TestError to get un-deployed
2021/12/10 18:54:14 App: TestError got un-deployed
2021/12/10 18:54:14 No eventing-con process running
2021/12/10 18:54:14 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/10 18:54:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:54:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:54:23 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:54:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:54:28 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:54:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/10 18:54:34 Waiting for app: TestError to get deployed
2021/12/10 18:54:39 Waiting for app: TestError to get deployed
2021/12/10 18:54:44 Waiting for app: TestError to get deployed
2021/12/10 18:54:49 Waiting for app: TestError to get deployed
2021/12/10 18:54:49 App: TestError got deployed
2021/12/10 18:54:49 Starting bucket ops to source bucket
2021/12/10 18:54:54 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/10 18:54:59 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/10 18:54:59 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/10 18:54:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:54:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:55:04 Waiting for app: TestError to get un-deployed
2021/12/10 18:55:09 Waiting for app: TestError to get un-deployed
2021/12/10 18:55:14 Waiting for app: TestError to get un-deployed
2021/12/10 18:55:19 Waiting for app: TestError to get un-deployed
2021/12/10 18:55:19 App: TestError got un-deployed
2021/12/10 18:55:19 No eventing-con process running
2021/12/10 18:55:19 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/10 18:55:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:55:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:55:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:55:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:55:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:55:48 Waiting for dst bucket item count to get to: 0 curr count: 183
2021/12/10 18:55:53 Waiting for dst bucket item count to get to: 0 curr count: 39
2021/12/10 18:55:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.67s)
=== RUN   TestCRLF
2021/12/10 18:56:03 Waiting for app: TestCRLF to get un-deployed
2021/12/10 18:56:03 App: TestCRLF got un-deployed
2021/12/10 18:56:03 No eventing-con process running
2021/12/10 18:56:03 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/10 18:56:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:56:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:56:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:56:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:56:16 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:56:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/10 18:56:23 Waiting for app: TestCRLF to get deployed
2021/12/10 18:56:28 Waiting for app: TestCRLF to get deployed
2021/12/10 18:56:33 Waiting for app: TestCRLF to get deployed
2021/12/10 18:56:38 Waiting for app: TestCRLF to get deployed
2021/12/10 18:56:38 App: TestCRLF got deployed
2021/12/10 18:56:38 Starting bucket ops to source bucket
2021/12/10 18:56:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 18:56:48 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 18:56:53 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/10 18:56:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 18:56:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 18:56:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:56:58 Waiting for app: TestCRLF to get un-deployed
2021/12/10 18:57:03 Waiting for app: TestCRLF to get un-deployed
2021/12/10 18:57:08 Waiting for app: TestCRLF to get un-deployed
2021/12/10 18:57:13 Waiting for app: TestCRLF to get un-deployed
2021/12/10 18:57:13 App: TestCRLF got un-deployed
2021/12/10 18:57:13 No eventing-con process running
2021/12/10 18:57:13 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/10 18:57:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:57:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:57:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:57:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:57:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 18:57:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (103.61s)
=== RUN   TestImportExport
2021/12/10 18:57:46 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 18:57:47 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/10 18:57:47 No eventing-con process running
2021/12/10 18:57:47 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/10 18:57:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:57:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:57:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:58:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:58:05 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 18:58:05 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/10 18:58:05 No eventing-con process running
2021/12/10 18:58:05 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/10 18:58:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:58:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:58:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 18:58:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 18:58:19 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:58:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 18:58:20 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 18:58:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 18:58:26 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 18:58:31 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 18:58:36 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 18:58:41 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/10 18:58:41 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/10 18:58:46 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/10 18:58:51 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/10 18:58:56 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/10 18:58:56 App: TestImportExport_bucket_op_on_update got deployed
2021/12/10 18:58:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:59:01 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 18:59:06 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 18:59:11 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 18:59:16 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 18:59:16 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/10 18:59:16 No eventing-con process running
2021/12/10 18:59:16 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/10 18:59:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 18:59:21 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 18:59:26 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 18:59:31 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 18:59:36 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 18:59:36 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/10 18:59:36 No eventing-con process running
2021/12/10 18:59:36 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/10 18:59:54 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/10 18:59:54 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/10 18:59:54 No eventing-con process running
2021/12/10 18:59:54 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/10 18:59:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:00:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:00:02 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:00:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:00:12 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/10 19:00:12 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/10 19:00:12 No eventing-con process running
2021/12/10 19:00:12 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/10 19:00:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:00:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:00:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:00:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.31s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/10 19:00:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:00:36 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 19:00:36 No eventing-con process running
2021/12/10 19:00:36 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/10 19:00:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:00:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:00:44 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:00:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:00:49 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:00:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 19:00:50 Starting bucket ops to source bucket
2021/12/10 19:00:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:01:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:01:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:01:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:01:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:01:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 19:01:19 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 19:01:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:01:19 Undeploying app: bucket_op_on_update
2021/12/10 19:01:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:01:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:01:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:01:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:01:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:01:39 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 19:01:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:01:39 No eventing-con process running
2021/12/10 19:01:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:01:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:02:23 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:02:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 19:02:24 Starting bucket ops to source bucket
2021/12/10 19:02:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:02:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:02:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:02:47 Waiting for dst bucket item count to get to: 5000 curr count: 1571
2021/12/10 19:02:52 Waiting for dst bucket item count to get to: 5000 curr count: 4071
2021/12/10 19:02:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:02:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 19:02:57 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 19:02:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:02:57 Undeploying app: bucket_op_on_update
2021/12/10 19:02:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:03:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:03:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:03:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:03:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:03:17 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 19:03:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:03:17 No eventing-con process running
2021/12/10 19:03:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:03:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:04:01 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:04:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 19:04:02 Starting bucket ops to source bucket
2021/12/10 19:04:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:04:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:04:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:04:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:04:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:04:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 19:04:30 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 19:04:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:04:30 Undeploying app: bucket_op_on_update
2021/12/10 19:04:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:04:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:04:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:04:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:04:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:04:50 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 19:04:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:04:50 No eventing-con process running
2021/12/10 19:04:52 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:04:59 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:05:34 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:05:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 19:05:36 Starting bucket ops to source bucket
2021/12/10 19:05:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:05:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:05:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:05:59 Waiting for dst bucket item count to get to: 5000 curr count: 2585
2021/12/10 19:06:04 Waiting for dst bucket item count to get to: 5000 curr count: 2585
2021/12/10 19:06:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:06:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 19:06:09 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 19:06:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:06:09 Undeploying app: bucket_op_on_update
2021/12/10 19:06:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:06:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:06:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:06:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:06:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:06:29 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 19:06:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:06:29 No eventing-con process running
2021/12/10 19:06:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:06:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:07:13 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:07:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 19:07:14 Starting bucket ops to source bucket
2021/12/10 19:07:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:07:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:07:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:07:37 Waiting for dst bucket item count to get to: 5000 curr count: 1553
2021/12/10 19:07:42 Waiting for dst bucket item count to get to: 5000 curr count: 4053
2021/12/10 19:07:42 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:07:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/10 19:07:47 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/10 19:07:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:07:47 Undeploying app: bucket_op_on_update
2021/12/10 19:07:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:07:52 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:07:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:08:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:08:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/10 19:08:07 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/10 19:08:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:08:07 No eventing-con process running
2021/12/10 19:08:09 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:08:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:08:51 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (505.21s)
=== RUN   TestOnUpdateN1QLOp
2021/12/10 19:09:01 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 19:09:01 App: TestOnUpdateN1QLOp got un-deployed
2021/12/10 19:09:01 No eventing-con process running
2021/12/10 19:09:01 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/10 19:09:03 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:09:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:09:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:09:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:09:15 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:09:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/10 19:09:16 Starting bucket ops to source bucket
2021/12/10 19:09:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:09:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:09:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:09:39 Waiting for dst bucket item count to get to: 5000 curr count: 975
2021/12/10 19:09:44 Waiting for dst bucket item count to get to: 5000 curr count: 975
2021/12/10 19:09:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:09:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:09:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:09:49 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 19:09:54 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 19:09:59 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 19:10:04 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/10 19:10:04 App: TestOnUpdateN1QLOp got un-deployed
2021/12/10 19:10:04 No eventing-con process running
2021/12/10 19:10:04 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/10 19:10:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:10:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:10:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:10:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:10:28 Waiting for dst bucket item count to get to: 0 curr count: 2464
2021/12/10 19:10:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (96.60s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/10 19:10:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 19:10:38 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/10 19:10:38 No eventing-con process running
2021/12/10 19:10:38 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/10 19:10:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:10:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:10:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:10:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:10:51 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:10:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 19:10:52 Starting bucket ops to source bucket
2021/12/10 19:11:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:11:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:11:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:11:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:11:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:11:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:11:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:11:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 19:11:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 19:11:30 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 19:11:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/10 19:11:35 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/10 19:11:35 No eventing-con process running
2021/12/10 19:11:35 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/10 19:11:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:11:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:11:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:11:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:11:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:12:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 19:12:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.32s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/10 19:12:14 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 19:12:14 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 19:12:14 No eventing-con process running
2021/12/10 19:12:14 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/10 19:12:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:12:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:12:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:12:27 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:12:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 19:12:29 Starting bucket ops to source bucket
2021/12/10 19:12:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:12:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:12:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:12:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:12:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 19:12:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:12:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:12:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:13:02 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 19:13:07 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 19:13:12 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 19:13:17 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 19:13:17 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 19:13:17 No eventing-con process running
2021/12/10 19:13:17 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/10 19:13:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:13:29 Waiting for dst bucket item count to get to: 0 curr count: 4382
2021/12/10 19:13:34 Waiting for dst bucket item count to get to: 0 curr count: 1882
2021/12/10 19:13:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:13:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:13:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:13:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.39s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/10 19:13:55 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 19:13:55 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/10 19:13:55 No eventing-con process running
2021/12/10 19:13:55 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/10 19:13:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:14:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:14:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:14:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:14:09 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:14:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/10 19:14:10 Starting bucket ops to source bucket
2021/12/10 19:14:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:14:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:14:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:14:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:14:38 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 19:14:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:14:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:14:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:14:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 19:14:48 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 19:14:53 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 19:14:58 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/10 19:14:58 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/10 19:14:58 No eventing-con process running
2021/12/10 19:14:58 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/10 19:15:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:15:10 Waiting for dst bucket item count to get to: 0 curr count: 3205
2021/12/10 19:15:15 Waiting for dst bucket item count to get to: 0 curr count: 3205
2021/12/10 19:15:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:15:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:15:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:15:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.60s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/10 19:15:37 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 19:15:37 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/10 19:15:37 No eventing-con process running
2021/12/10 19:15:37 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/10 19:15:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:15:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:15:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:15:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:15:51 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:15:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/10 19:15:52 Starting bucket ops to source bucket
2021/12/10 19:16:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:16:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:16:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:16:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:16:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:16:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:16:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:16:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 19:16:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 19:16:30 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 19:16:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/10 19:16:35 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/10 19:16:35 No eventing-con process running
2021/12/10 19:16:35 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/10 19:16:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:16:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:16:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:16:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:16:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:17:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 19:17:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (96.96s)
=== RUN   TestOnDeleteBucketOp
2021/12/10 19:17:14 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 19:17:14 App: TestOnDeleteBucketOp got un-deployed
2021/12/10 19:17:14 No eventing-con process running
2021/12/10 19:17:14 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/10 19:17:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:17:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:17:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:17:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:17:27 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:17:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 19:17:29 Starting bucket ops to source bucket
2021/12/10 19:17:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:17:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:17:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:17:54 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/10 19:17:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:17:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:17:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:17:59 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 19:18:04 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 19:18:09 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 19:18:14 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/10 19:18:14 App: TestOnDeleteBucketOp got un-deployed
2021/12/10 19:18:14 No eventing-con process running
2021/12/10 19:18:14 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/10 19:18:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:18:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:18:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:18:34 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:18:39 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/10 19:18:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (94.74s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/10 19:18:49 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 19:18:49 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/10 19:18:49 No eventing-con process running
2021/12/10 19:18:49 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/10 19:18:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:18:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:18:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:19:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:19:02 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:19:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/10 19:19:03 Starting bucket ops to source bucket
2021/12/10 19:19:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:19:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:19:24 Waiting for dst bucket item count to get to: 5000 curr count: 2428
2021/12/10 19:19:29 Waiting for dst bucket item count to get to: 5000 curr count: 4924
2021/12/10 19:19:29 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:19:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:19:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:19:34 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 19:19:39 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 19:19:44 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 19:19:49 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/10 19:19:49 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/10 19:19:49 No eventing-con process running
2021/12/10 19:19:49 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/10 19:19:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:19:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:19:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:20:08 Waiting for dst bucket item count to get to: 0 curr count: 3443
2021/12/10 19:20:13 Waiting for dst bucket item count to get to: 0 curr count: 947
2021/12/10 19:20:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (94.64s)
=== RUN   TestTimerBucketOp
2021/12/10 19:20:23 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 19:20:23 App: TestTimerBucketOp got un-deployed
2021/12/10 19:20:23 No eventing-con process running
2021/12/10 19:20:23 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/10 19:20:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:20:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:20:32 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:20:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:20:37 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:20:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 19:20:38 Starting bucket ops to source bucket
2021/12/10 19:20:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:20:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:20:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:21:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:21:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:21:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:21:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:21:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:21:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:21:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:21:21 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 19:21:26 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 19:21:31 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 19:21:36 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/10 19:21:36 App: TestTimerBucketOp got un-deployed
2021/12/10 19:21:36 No eventing-con process running
2021/12/10 19:21:36 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/10 19:21:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:21:48 Waiting for dst bucket item count to get to: 0 curr count: 2851
2021/12/10 19:21:53 Waiting for dst bucket item count to get to: 0 curr count: 351
2021/12/10 19:21:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:21:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:22:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:22:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (111.28s)
=== RUN   TestTimerInPastBucketOp
2021/12/10 19:22:15 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 19:22:15 App: TestTimerInPastBucketOp got un-deployed
2021/12/10 19:22:15 No eventing-con process running
2021/12/10 19:22:15 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/10 19:22:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:22:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:22:23 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:22:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:22:28 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:22:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 19:22:29 Starting bucket ops to source bucket
2021/12/10 19:22:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:22:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:22:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:22:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:22:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:23:02 Waiting for dst bucket item count to get to: 5000 curr count: 107
2021/12/10 19:23:07 Waiting for dst bucket item count to get to: 5000 curr count: 1905
2021/12/10 19:23:12 Waiting for dst bucket item count to get to: 5000 curr count: 4298
2021/12/10 19:23:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:23:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:23:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:23:17 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 19:23:22 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 19:23:27 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 19:23:32 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/10 19:23:32 App: TestTimerInPastBucketOp got un-deployed
2021/12/10 19:23:32 No eventing-con process running
2021/12/10 19:23:32 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/10 19:23:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:23:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:23:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 19:23:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:23:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:24:01 Waiting for dst bucket item count to get to: 0 curr count: 4713
2021/12/10 19:24:06 Waiting for dst bucket item count to get to: 0 curr count: 4713
2021/12/10 19:24:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (121.30s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/10 19:24:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:24:16 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 19:24:16 No eventing-con process running
2021/12/10 19:24:16 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/10 19:24:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:24:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:24:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:24:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:24:29 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:24:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 19:24:30 Starting bucket ops to source bucket
2021/12/10 19:24:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:24:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:24:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:24:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:24:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:25:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:25:09 Waiting for dst bucket item count to get to: 5000 curr count: 2877
2021/12/10 19:25:14 Waiting for dst bucket item count to get to: 5000 curr count: 2877
2021/12/10 19:25:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:25:19 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 19:25:19 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 19:25:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:25:19 Undeploying app: bucket_op_with_timer
2021/12/10 19:25:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:25:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:25:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:25:34 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:25:39 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:25:39 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 19:25:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:25:39 No eventing-con process running
2021/12/10 19:25:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:25:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:26:23 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:26:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 19:26:24 Starting bucket ops to source bucket
2021/12/10 19:26:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:26:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:26:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:26:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:26:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:26:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:27:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:27:07 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 19:27:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:27:12 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 19:27:12 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 19:27:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:27:12 Undeploying app: bucket_op_with_timer
2021/12/10 19:27:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:27:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:27:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:27:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:27:32 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:27:32 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 19:27:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:27:32 No eventing-con process running
2021/12/10 19:27:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:27:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:28:16 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:28:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 19:28:17 Starting bucket ops to source bucket
2021/12/10 19:28:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:28:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:28:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:28:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:28:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:28:50 Waiting for dst bucket item count to get to: 5000 curr count: 391
2021/12/10 19:28:55 Waiting for dst bucket item count to get to: 5000 curr count: 391
2021/12/10 19:29:00 Waiting for dst bucket item count to get to: 5000 curr count: 3888
2021/12/10 19:29:05 Waiting for dst bucket item count to get to: 5000 curr count: 3888
2021/12/10 19:29:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:29:10 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 19:29:10 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 19:29:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:29:10 Undeploying app: bucket_op_with_timer
2021/12/10 19:29:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:29:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:29:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:29:25 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:29:30 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:29:30 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 19:29:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:29:30 No eventing-con process running
2021/12/10 19:29:32 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:29:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:30:14 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:30:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 19:30:15 Starting bucket ops to source bucket
2021/12/10 19:30:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:30:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:30:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:30:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:30:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:30:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:30:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:30:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:30:58 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 19:30:58 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 19:30:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:30:58 Undeploying app: bucket_op_with_timer
2021/12/10 19:30:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:31:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:31:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:31:14 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:31:19 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:31:19 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 19:31:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:31:19 No eventing-con process running
2021/12/10 19:31:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:31:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:32:03 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:32:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 19:32:04 Starting bucket ops to source bucket
2021/12/10 19:32:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:32:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:32:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:32:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:32:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:32:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:32:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:32:47 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 19:32:47 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:32:52 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/10 19:32:52 App: TestDeployUndeployLoopTimer got deployed
2021/12/10 19:32:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:32:52 Undeploying app: bucket_op_with_timer
2021/12/10 19:32:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:32:57 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:33:02 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:33:07 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:33:12 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/10 19:33:12 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/10 19:33:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:33:12 No eventing-con process running
2021/12/10 19:33:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:33:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:33:56 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (589.98s)
=== RUN   TestMultipleHandlers
2021/12/10 19:34:06 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 19:34:06 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/10 19:34:06 No eventing-con process running
2021/12/10 19:34:06 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/10 19:34:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:34:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:34:14 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:34:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:34:24 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 19:34:24 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/10 19:34:24 No eventing-con process running
2021/12/10 19:34:24 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/10 19:34:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:34:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:34:33 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:34:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:34:38 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:34:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 19:34:39 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:34:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/10 19:34:45 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/10 19:34:50 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/10 19:34:55 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/10 19:35:00 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/10 19:35:00 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/10 19:35:05 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/10 19:35:10 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/10 19:35:15 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/10 19:35:15 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/10 19:35:15 Starting bucket ops to source bucket
2021/12/10 19:35:25 Waiting for dst bucket item count to get to: 5000 curr count: 965
2021/12/10 19:35:30 Waiting for dst bucket item count to get to: 5000 curr count: 2985
2021/12/10 19:35:35 Waiting for dst bucket item count to get to: 5000 curr count: 4520
2021/12/10 19:35:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:35:35 Starting bucket ops to source bucket
2021/12/10 19:35:50 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:35:55 Waiting for dst bucket item count to get to: 10000 curr count: 7504
2021/12/10 19:35:55 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 19:35:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:35:55 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:35:55 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:35:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:36:01 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 19:36:06 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 19:36:11 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 19:36:16 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/10 19:36:16 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/10 19:36:16 No eventing-con process running
2021/12/10 19:36:16 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/10 19:36:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:36:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:36:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:36:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 19:36:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:36:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:36:40 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 19:36:45 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 19:36:50 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 19:36:55 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/10 19:36:55 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/10 19:36:55 No eventing-con process running
2021/12/10 19:36:55 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/10 19:36:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:37:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:37:03 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:37:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (192.47s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/10 19:37:18 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 19:37:18 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/10 19:37:18 No eventing-con process running
2021/12/10 19:37:18 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/10 19:37:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:37:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:37:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:37:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:37:32 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:37:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 19:37:33 Starting bucket ops to source bucket
2021/12/10 19:37:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:37:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:37:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:37:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:37:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:37:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:38:01 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 19:38:01 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 19:38:01 Pausing app: bucket_op_on_update
2021/12/10 19:38:01 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:38:06 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 19:38:06 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 19:38:06 Resuming app: bucket_op_on_update
2021/12/10 19:38:07 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:38:07 Starting bucket ops to source bucket
2021/12/10 19:38:15 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:38:20 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:38:25 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/10 19:38:25 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 19:38:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:38:30 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 19:38:30 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 19:38:30 Pausing app: bucket_op_on_update
2021/12/10 19:38:30 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:38:35 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 19:38:35 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 19:38:35 Resuming app: bucket_op_on_update
2021/12/10 19:38:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:38:37 Starting bucket ops to source bucket
2021/12/10 19:38:44 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 19:38:49 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 19:38:54 Waiting for dst bucket item count to get to: 15000 curr count: 12500
2021/12/10 19:38:54 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/10 19:38:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:38:59 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 19:38:59 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 19:38:59 Pausing app: bucket_op_on_update
2021/12/10 19:39:00 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:39:05 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 19:39:05 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 19:39:05 Resuming app: bucket_op_on_update
2021/12/10 19:39:06 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:39:06 Starting bucket ops to source bucket
2021/12/10 19:39:14 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 19:39:19 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 19:39:24 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 19:39:29 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/10 19:39:29 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/10 19:39:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:39:34 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 19:39:34 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 19:39:34 Pausing app: bucket_op_on_update
2021/12/10 19:39:34 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:39:39 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 19:39:39 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 19:39:39 Resuming app: bucket_op_on_update
2021/12/10 19:39:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:39:40 Starting bucket ops to source bucket
2021/12/10 19:39:48 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 19:39:53 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 19:39:58 Waiting for dst bucket item count to get to: 25000 curr count: 21163
2021/12/10 19:40:03 Waiting for dst bucket item count to get to: 25000 curr count: 23663
2021/12/10 19:40:03 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/10 19:40:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:40:08 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/10 19:40:08 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/10 19:40:08 Pausing app: bucket_op_on_update
2021/12/10 19:40:08 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:40:13 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/10 19:40:13 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/10 19:40:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:40:18 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 19:40:23 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 19:40:28 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 19:40:33 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/10 19:40:33 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/10 19:40:33 No eventing-con process running
2021/12/10 19:40:33 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/10 19:40:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:40:45 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 19:40:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:40:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:40:57 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 19:40:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (228.61s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/10 19:41:07 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 19:41:07 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/10 19:41:07 No eventing-con process running
2021/12/10 19:41:07 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/10 19:41:09 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:41:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:41:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:41:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:41:20 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:41:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 19:41:22 Starting bucket ops to source bucket
2021/12/10 19:41:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:41:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:41:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:41:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:41:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:41:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:41:59 Waiting for dst bucket item count to get to: 5000 curr count: 3548
2021/12/10 19:42:04 Waiting for dst bucket item count to get to: 5000 curr count: 3548
2021/12/10 19:42:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:42:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:42:09 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 19:42:10 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 19:42:10 Pausing app: bucket_op_with_timer
2021/12/10 19:42:10 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:42:15 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 19:42:15 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 19:42:15 Resuming app: bucket_op_with_timer
2021/12/10 19:42:16 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:42:16 Starting bucket ops to source bucket
2021/12/10 19:42:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:42:29 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:42:34 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:42:39 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:42:44 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:42:49 Waiting for dst bucket item count to get to: 10000 curr count: 8354
2021/12/10 19:42:54 Waiting for dst bucket item count to get to: 10000 curr count: 8354
2021/12/10 19:42:54 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 19:42:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:42:59 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 19:42:59 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 19:42:59 Pausing app: bucket_op_with_timer
2021/12/10 19:42:59 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:43:04 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 19:43:04 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 19:43:04 Resuming app: bucket_op_with_timer
2021/12/10 19:43:05 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:43:05 Starting bucket ops to source bucket
2021/12/10 19:43:13 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 19:43:18 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 19:43:23 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/10 19:43:28 Waiting for dst bucket item count to get to: 15000 curr count: 10048
2021/12/10 19:43:33 Waiting for dst bucket item count to get to: 15000 curr count: 10048
2021/12/10 19:43:38 Waiting for dst bucket item count to get to: 15000 curr count: 14280
2021/12/10 19:43:38 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/10 19:43:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:43:43 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 19:43:43 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 19:43:43 Pausing app: bucket_op_with_timer
2021/12/10 19:43:43 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:43:48 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 19:43:48 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 19:43:48 Resuming app: bucket_op_with_timer
2021/12/10 19:43:49 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:43:49 Starting bucket ops to source bucket
2021/12/10 19:43:57 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 19:44:02 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 19:44:07 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 19:44:12 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 19:44:17 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 19:44:22 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/10 19:44:27 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/10 19:44:27 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/10 19:44:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:44:32 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 19:44:32 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 19:44:32 Pausing app: bucket_op_with_timer
2021/12/10 19:44:33 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:44:38 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 19:44:38 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 19:44:38 Resuming app: bucket_op_with_timer
2021/12/10 19:44:39 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:44:39 Starting bucket ops to source bucket
2021/12/10 19:44:47 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 19:44:52 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 19:44:57 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 19:45:02 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 19:45:07 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 19:45:12 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/10 19:45:17 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/10 19:45:17 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/10 19:45:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:45:22 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/10 19:45:22 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/10 19:45:22 Pausing app: bucket_op_with_timer
2021/12/10 19:45:22 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:45:27 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/10 19:45:27 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/10 19:45:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:45:32 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 19:45:37 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 19:45:42 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 19:45:47 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/10 19:45:47 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/10 19:45:47 No eventing-con process running
2021/12/10 19:45:47 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/10 19:45:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:45:59 Waiting for dst bucket item count to get to: 0 curr count: 23395
2021/12/10 19:46:04 Waiting for dst bucket item count to get to: 0 curr count: 10895
2021/12/10 19:46:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:46:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:46:15 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 19:46:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (318.52s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/10 19:46:25 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 19:46:25 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/10 19:46:25 No eventing-con process running
2021/12/10 19:46:25 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/10 19:46:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:46:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:46:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:46:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:46:39 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:46:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/10 19:46:45 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 19:46:50 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 19:46:55 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 19:47:00 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/10 19:47:00 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/10 19:47:00 Starting bucket ops to source bucket
2021/12/10 19:47:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:47:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:47:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:47:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:47:29 Waiting for dst bucket item count to get to: 5000 curr count: 3953
2021/12/10 19:47:34 Waiting for dst bucket item count to get to: 5000 curr count: 3953
2021/12/10 19:47:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:47:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:47:34 Pausing app: bucket_op_with_timer
2021/12/10 19:47:34 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:47:39 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/10 19:47:40 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/10 19:47:40 Resuming app: bucket_op_with_timer
2021/12/10 19:47:41 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:47:41 Starting bucket ops to source bucket
2021/12/10 19:47:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:47:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:48:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:48:07 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:48:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:48:12 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 19:48:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:48:17 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 19:48:22 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 19:48:27 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 19:48:32 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/10 19:48:32 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/10 19:48:32 No eventing-con process running
2021/12/10 19:48:32 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/10 19:48:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:48:44 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 19:48:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:48:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:48:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:49:01 Waiting for dst bucket item count to get to: 0 curr count: 9665
2021/12/10 19:49:06 Waiting for dst bucket item count to get to: 0 curr count: 9665
2021/12/10 19:49:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (170.48s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/10 19:49:16 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 19:49:16 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/10 19:49:16 No eventing-con process running
2021/12/10 19:49:16 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/10 19:49:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:49:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:49:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:49:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:49:29 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:49:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/10 19:49:36 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 19:49:41 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 19:49:46 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 19:49:51 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/10 19:49:51 App: TestPauseResumeWithEventingReb got deployed
2021/12/10 19:49:51 Starting bucket ops to source bucket
2021/12/10 19:50:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:50:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:50:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:50:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:50:20 Waiting for dst bucket item count to get to: 5000 curr count: 930
2021/12/10 19:50:25 Waiting for dst bucket item count to get to: 5000 curr count: 930
2021/12/10 19:50:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:50:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:50:25 Pausing app: bucket_op_with_timer
2021/12/10 19:50:25 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:50:30 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/10 19:50:30 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/10 19:50:30 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 19:50:30 Removing node(s): [] from the cluster
2021/12/10 19:50:35 Rebalance progress: 29.16666666666666
2021/12/10 19:50:40 Rebalance progress: 29.16666666666666
2021/12/10 19:50:45 Rebalance progress: 100
2021/12/10 19:50:45 Resuming app: bucket_op_with_timer
2021/12/10 19:50:46 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:50:46 Starting bucket ops to source bucket
2021/12/10 19:50:58 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:51:03 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:51:08 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:51:13 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/10 19:51:18 Waiting for dst bucket item count to get to: 10000 curr count: 6383
2021/12/10 19:51:23 Waiting for dst bucket item count to get to: 10000 curr count: 6383
2021/12/10 19:51:23 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 19:51:23 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 19:51:28 Rebalance progress: 29.16666666666666
2021/12/10 19:51:33 Rebalance progress: 29.16666666666666
2021/12/10 19:51:38 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/10 19:51:43 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:83} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 19:51:58 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:83} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 19:52:13 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:83} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 19:52:28 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:83} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 19:52:43 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:83} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 19:52:58 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:83} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 19:53:13 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:83} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 19:53:28 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:83} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 19:53:43 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:83} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 19:53:58 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:83} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: 798fe84c93f474ef53f28c37b42bb067
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
Producer node: 127.0.0.1:9302	Node UUID: 5367ac45a03a04ae59b1cfdb9db613b1
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
Producer node: 172.31.2.134:9300	Node UUID: 247fa8fff8833e51f34d05319a3540a5
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]


2021/12/10 19:54:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:54:13 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 19:54:18 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 19:54:23 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 19:54:28 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 19:54:33 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/10 19:54:33 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/10 19:54:33 eventingSetup.go:46: No eventing-con process running
2021/12/10 19:54:33 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/10 19:54:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:54:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 19:54:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:54:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:54:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 19:54:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (351.12s)
=== RUN   TestBucketDeleteAfterPause
2021/12/10 19:55:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/10 19:55:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/10 19:55:07 eventingSetup.go:46: No eventing-con process running
2021/12/10 19:55:07 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/10 19:55:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:55:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:55:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:55:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:55:21 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:55:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/10 19:55:27 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 19:55:32 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 19:55:37 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 19:55:42 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/10 19:55:42 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/10 19:55:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 19:55:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:55:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 19:55:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 19:55:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 19:55:56 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/10 19:55:57 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:56:02 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/10 19:56:02 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/10 19:56:02 handler_test.go:716: Deleting source bucket: default
2021/12/10 19:56:08 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/10 19:56:08 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/10 19:56:13 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/10 19:56:13 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/10 19:56:13 eventingSetup.go:46: No eventing-con process running
2021/12/10 19:56:13 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/10 19:56:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:56:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 19:56:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:56:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:56:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 19:56:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (99.11s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/10 19:56:46 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 19:56:46 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/10 19:56:46 eventingSetup.go:46: No eventing-con process running
2021/12/10 19:56:46 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/10 19:56:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:56:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:56:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 19:56:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 19:56:59 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/10 19:57:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/10 19:57:05 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 19:57:10 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 19:57:15 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 19:57:20 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/10 19:57:20 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/10 19:57:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 19:57:55 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/10 19:57:56 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 19:58:01 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/10 19:58:01 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/10 20:01:01 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/10 20:01:01 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/10 20:01:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/10 20:01:07 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 20:01:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:01:07 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/10 20:01:12 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 20:01:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:01:12 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2870 expected 5000
2021/12/10 20:01:17 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 20:01:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:01:17 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 4449 expected 5000
2021/12/10 20:01:22 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/10 20:01:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:01:22 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/10 20:01:22 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/10 20:01:22 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 20:01:32 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 20:01:37 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 20:01:42 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/10 20:01:42 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/10 20:01:42 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:01:42 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/10 20:01:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:01:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:01:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 20:01:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:02:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:02:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.72s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/10 20:02:15 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 20:02:15 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/10 20:02:15 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:02:15 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/10 20:02:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:02:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:02:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:02:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:02:28 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:02:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 20:02:34 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 20:02:39 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 20:02:44 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 20:02:49 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 20:02:49 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/10 20:02:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 20:02:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:03:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 76
2021/12/10 20:03:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2576
2021/12/10 20:03:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 20:03:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:03:08 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/10 20:03:08 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 20:03:38 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/10 20:03:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 20:03:44 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 20:03:49 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 20:03:54 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 20:03:59 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/10 20:03:59 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/10 20:03:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 20:04:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:04:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:04:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:04:12 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/10 20:04:12 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 20:04:22 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 20:04:27 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 20:04:32 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/10 20:04:32 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/10 20:04:32 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:04:32 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/10 20:04:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:04:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:04:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:04:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (160.33s)
=== RUN   TestCPPWorkerCleanup
2021/12/10 20:04:55 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 20:04:55 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/10 20:04:55 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:04:55 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/10 20:04:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:05:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:05:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:05:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:05:08 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:05:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/10 20:05:09 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 20:05:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:05:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:05:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:05:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:05:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 20:05:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 20:05:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:05:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:05:42 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 20:05:47 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 20:05:52 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 20:05:57 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/10 20:05:57 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/10 20:05:57 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:05:57 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/10 20:05:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:06:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4032
2021/12/10 20:06:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1532
2021/12/10 20:06:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:06:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:06:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:06:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (129.64s)
=== RUN   TestWithUserXattrs
2021/12/10 20:07:05 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 20:07:05 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/10 20:07:05 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:07:05 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/10 20:07:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:07:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:07:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:07:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:07:18 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 20:07:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 20:07:22 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:07:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/10 20:07:28 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 20:07:33 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 20:07:38 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 20:07:43 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/10 20:07:43 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/10 20:07:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 20:07:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:07:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 20:07:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 20:07:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:07:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:08:02 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 20:08:07 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 20:08:12 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 20:08:17 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/10 20:08:17 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/10 20:08:17 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:08:17 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/10 20:08:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:08:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:08:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 20:08:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:08:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:08:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:08:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (110.55s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/10 20:08:55 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 20:08:55 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/10 20:08:55 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:08:55 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/10 20:08:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:09:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:09:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:09:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:09:08 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:09:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/10 20:09:15 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 20:09:20 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 20:09:25 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 20:09:30 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/10 20:09:30 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/10 20:09:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 20:09:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:09:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:09:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/10 20:09:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 20:09:49 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:09:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:10:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4446
2021/12/10 20:10:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1946
2021/12/10 20:10:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:10:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:10:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:10:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 20:10:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:10:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 20:10:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 17885
2021/12/10 20:10:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 17885
2021/12/10 20:10:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/10 20:10:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:10:54 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 20:10:59 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 20:11:04 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 20:11:09 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 20:11:14 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 20:11:14 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/10 20:11:14 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:11:19 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/10 20:11:19 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/10 20:11:19 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:11:19 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/10 20:11:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:11:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 23203
2021/12/10 20:11:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 23203
2021/12/10 20:11:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:11:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:11:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/10 20:11:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (181.71s)
=== RUN   TestMetaBucketDelete
2021/12/10 20:11:57 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/10 20:11:57 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/10 20:11:57 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:11:57 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/10 20:11:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:12:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:12:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:12:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:12:10 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:12:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 20:12:16 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 20:12:21 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 20:12:26 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 20:12:31 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/10 20:12:31 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/10 20:12:31 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/10 20:12:33 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/10 20:12:38 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/10 20:12:38 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/10 20:12:53 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/10 20:12:53 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/10 20:12:53 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:12:53 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/10 20:12:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:12:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:13:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:13:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.95s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/10 20:13:16 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 20:13:16 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 20:13:16 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:13:16 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/10 20:13:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:13:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:13:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:13:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:13:29 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:13:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 20:13:49 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/10 20:13:50 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/10 20:14:05 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 20:14:05 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 20:14:20 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 20:14:20 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 20:14:20 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:14:20 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/10 20:14:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:14:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:14:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:14:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.65s)
=== RUN   TestSourceBucketDelete
2021/12/10 20:14:43 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/10 20:14:43 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/10 20:14:43 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:14:43 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/10 20:14:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:14:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:14:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:14:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:14:56 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:14:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/10 20:15:02 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 20:15:07 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 20:15:12 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 20:15:17 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/10 20:15:17 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/10 20:15:17 handler_test.go:945: Deleting source bucket: default
2021/12/10 20:15:19 handler_test.go:947: Deleted source bucket: default
2021/12/10 20:15:24 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/10 20:15:24 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/10 20:15:39 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/10 20:15:39 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/10 20:15:39 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:15:39 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/10 20:15:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:15:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:15:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:15:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.11s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/10 20:16:02 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/10 20:16:02 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/10 20:16:02 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:16:02 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/10 20:16:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:16:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:16:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:16:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:16:15 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:16:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 20:16:35 handler_test.go:966: Deleting source bucket: default
2021/12/10 20:16:36 handler_test.go:968: Deleted source bucket: default
2021/12/10 20:16:51 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/10 20:16:51 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/10 20:17:06 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/10 20:17:06 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/10 20:17:06 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:17:06 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/10 20:17:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:17:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:17:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:17:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.62s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/10 20:17:29 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 20:17:29 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 20:17:29 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:17:29 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/10 20:17:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:17:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:17:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:17:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:17:42 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:17:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 20:18:02 handler_test.go:988: Deleting source bucket: default
2021/12/10 20:18:04 handler_test.go:990: Deleted source bucket: default
2021/12/10 20:18:04 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/10 20:18:04 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/10 20:18:19 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 20:18:19 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 20:18:35 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/10 20:18:35 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/10 20:18:35 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:18:35 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/10 20:18:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:18:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:18:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:18:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.18s)
=== RUN   TestUndeployDuringBootstrap
2021/12/10 20:18:58 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 20:18:58 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/10 20:18:58 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:18:58 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/10 20:18:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:19:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:19:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:19:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:19:11 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:19:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 20:19:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 20:19:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:19:24 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:19:24 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/10 20:19:24 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/10 20:19:24 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/10 20:19:29 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/10 20:19:34 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/10 20:19:39 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 20:19:44 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 20:19:49 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 20:19:49 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/10 20:19:49 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:20:14 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/10 20:20:14 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/10 20:20:14 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:20:14 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/10 20:20:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:20:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:20:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:20:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:20:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3621
2021/12/10 20:20:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.42s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/10 20:20:42 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:20:43 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/10 20:20:48 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 20:20:53 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 20:20:58 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 20:21:03 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/10 20:21:03 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/10 20:21:03 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/10 20:21:03 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 20:21:03 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/10 20:21:08 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/10 20:21:08 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/10 20:21:08 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:21:08 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/10 20:21:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:21:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:21:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:21:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.55s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/10 20:21:32 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 20:21:32 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/10 20:21:32 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:21:32 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/10 20:21:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:21:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:21:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:21:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:21:45 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:21:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/10 20:21:51 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 20:21:56 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 20:22:01 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 20:22:06 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/10 20:22:06 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/10 20:22:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 20:22:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:22:41 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 20:22:46 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 20:22:51 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 20:22:56 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 20:22:56 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/10 20:22:56 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:24:41 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/10 20:24:41 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/10 20:24:41 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:24:41 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/10 20:24:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:24:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:24:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 20:24:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:25:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:25:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 20:25:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 20:25:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (232.99s)
=== RUN   TestUndeployWithKVFailover
2021/12/10 20:25:25 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/10 20:25:25 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/10 20:25:25 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:25:25 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/10 20:25:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:25:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:25:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:25:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:25:38 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:25:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/10 20:25:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 20:25:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:25:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:25:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:26:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:26:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:26:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:26:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2003
2021/12/10 20:26:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4503
2021/12/10 20:26:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 20:26:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 20:26:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 20:26:26 Rebalance progress: 29.16666666666666
2021/12/10 20:26:31 Rebalance progress: 29.16666666666666
2021/12/10 20:26:36 Rebalance progress: 100
2021/12/10 20:26:36 Failing over: [127.0.0.1:19003]
2021/12/10 20:26:46 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/10 20:27:46 Removing node(s): [] from the cluster
2021/12/10 20:27:51 Rebalance progress: 29.16666666666666
2021/12/10 20:27:56 Rebalance progress: 29.16666666666666
2021/12/10 20:28:01 Rebalance progress: 100
2021/12/10 20:28:06 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/10 20:28:06 App: TestUndeployWithKVFailover got un-deployed
2021/12/10 20:28:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:28:11 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/10 20:28:11 App: TestUndeployWithKVFailover got un-deployed
2021/12/10 20:28:11 No eventing-con process running
2021/12/10 20:28:12 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/10 20:28:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:28:23 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:28:28 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 20:28:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:28:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:28:40 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 20:28:45 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 20:28:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (210.02s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/10 20:28:55 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 20:28:55 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/10 20:28:55 No eventing-con process running
2021/12/10 20:28:55 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/10 20:28:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:29:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:29:03 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:29:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:29:08 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:29:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/10 20:29:14 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 20:29:19 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 20:29:24 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 20:29:29 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/10 20:29:29 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/10 20:29:29 Starting bucket ops to source bucket
2021/12/10 20:29:49 Waiting for dst bucket item count to get to: 20000 curr count: 12062
2021/12/10 20:29:54 Waiting for dst bucket item count to get to: 20000 curr count: 16738
2021/12/10 20:29:54 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/10 20:29:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:29:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:30:07 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/10 20:30:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:30:09 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:30:19 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/10 20:30:24 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 20:30:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:30:24 Starting bucket ops to source bucket
2021/12/10 20:30:37 Waiting for dst bucket item count to get to: 10000 curr count: 5346
2021/12/10 20:30:42 Waiting for dst bucket item count to get to: 10000 curr count: 5346
2021/12/10 20:30:42 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 20:30:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:30:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:30:47 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 20:30:52 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 20:30:57 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 20:31:02 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/10 20:31:02 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/10 20:31:02 No eventing-con process running
2021/12/10 20:31:02 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/10 20:31:03 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:31:13 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 20:31:18 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:31:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:31:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:31:30 Waiting for dst bucket item count to get to: 0 curr count: 6275
2021/12/10 20:31:35 Waiting for dst bucket item count to get to: 0 curr count: 6275
2021/12/10 20:31:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (170.40s)
=== RUN   TestTimerOverwrite
2021/12/10 20:31:45 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 20:31:45 App: TestTimerOverwrite got un-deployed
2021/12/10 20:31:45 No eventing-con process running
2021/12/10 20:31:45 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/10 20:31:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:31:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:31:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:31:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:31:58 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:31:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/10 20:32:04 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 20:32:09 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 20:32:14 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 20:32:19 Waiting for app: TestTimerOverwrite to get deployed
2021/12/10 20:32:19 App: TestTimerOverwrite got deployed
2021/12/10 20:32:29 Starting bucket ops to source bucket
2021/12/10 20:32:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:32:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:32:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:32:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:32:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:33:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:33:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:33:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:33:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:33:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 20:33:28 Waiting for dst bucket item count to get to: 5000 curr count: 346
2021/12/10 20:33:33 Waiting for dst bucket item count to get to: 5000 curr count: 669
2021/12/10 20:33:38 Waiting for dst bucket item count to get to: 5000 curr count: 2003
2021/12/10 20:33:43 Waiting for dst bucket item count to get to: 5000 curr count: 3937
2021/12/10 20:33:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 20:33:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:33:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:33:58 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 20:34:03 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 20:34:08 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 20:34:13 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/10 20:34:13 App: TestTimerOverwrite got un-deployed
2021/12/10 20:34:13 No eventing-con process running
2021/12/10 20:34:13 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/10 20:34:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:34:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:34:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:34:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:34:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:34:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (181.42s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/10 20:34:41 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 20:34:42 Removing node(s): [] from the cluster
2021/12/10 20:34:47 Rebalance progress: 100
2021/12/10 20:34:47 Failing over: [127.0.0.1:19003]
2021/12/10 20:34:52 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/10 20:34:52 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/10 20:34:52 No eventing-con process running
2021/12/10 20:34:52 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/10 20:34:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:34:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:35:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:35:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:35:05 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:35:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/10 20:35:06 Starting bucket ops to source bucket
2021/12/10 20:35:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:35:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:35:16 Context: Post to set retry counter, response code: 200 dump: 
2021/12/10 20:35:16 set retry counter, response &{[] }
2021/12/10 20:36:21 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/10 20:36:21 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/10 20:36:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:36:21 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/10 20:36:26 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/10 20:36:26 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/10 20:36:26 No eventing-con process running
2021/12/10 20:36:26 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/10 20:36:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:36:37 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 20:36:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:36:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:36:49 Waiting for dst bucket item count to get to: 0 curr count: 3910
2021/12/10 20:36:54 Waiting for dst bucket item count to get to: 0 curr count: 1410
2021/12/10 20:36:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:36:54 Removing node(s): [] from the cluster
2021/12/10 20:36:59 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (142.92s)
=== RUN   TestOnUpdateSrcMutation
2021/12/10 20:37:09 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 20:37:09 App: TestOnUpdateSrcMutation got un-deployed
2021/12/10 20:37:09 No eventing-con process running
2021/12/10 20:37:09 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/10 20:37:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:37:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:37:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:37:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:37:22 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:37:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/10 20:37:24 Starting bucket ops to source bucket
2021/12/10 20:37:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 20:37:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:37:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:37:51 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 20:37:56 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 20:38:01 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 20:38:06 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/10 20:38:06 App: TestOnUpdateSrcMutation got un-deployed
2021/12/10 20:38:06 No eventing-con process running
2021/12/10 20:38:06 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/10 20:38:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:38:17 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 20:38:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:38:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:38:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (84.40s)
=== RUN   TestOnDeleteSrcMutation
2021/12/10 20:38:34 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 20:38:34 App: TestOnDeleteSrcMutation got un-deployed
2021/12/10 20:38:34 No eventing-con process running
2021/12/10 20:38:34 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/10 20:38:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:38:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:38:42 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:38:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:38:47 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:38:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/10 20:38:48 Starting bucket ops to source bucket
2021/12/10 20:39:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 20:39:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:39:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:39:22 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 20:39:27 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 20:39:32 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 20:39:37 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/10 20:39:37 App: TestOnDeleteSrcMutation got un-deployed
2021/12/10 20:39:37 No eventing-con process running
2021/12/10 20:39:37 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/10 20:39:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:39:48 Waiting for dst bucket item count to get to: 0 curr count: 4566
2021/12/10 20:39:53 Waiting for dst bucket item count to get to: 0 curr count: 2066
2021/12/10 20:39:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:39:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:40:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.30s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/10 20:40:10 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 20:40:10 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/10 20:40:10 No eventing-con process running
2021/12/10 20:40:10 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/10 20:40:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:40:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:40:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:40:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:40:23 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:40:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 20:40:24 Starting bucket ops to source bucket
2021/12/10 20:41:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 20:41:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:41:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:41:11 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 20:41:17 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 20:41:22 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 20:41:27 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/10 20:41:27 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/10 20:41:27 No eventing-con process running
2021/12/10 20:41:27 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/10 20:41:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:41:38 Waiting for dst bucket item count to get to: 0 curr count: 6099
2021/12/10 20:41:43 Waiting for dst bucket item count to get to: 0 curr count: 1099
2021/12/10 20:41:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:41:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:41:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (109.53s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/10 20:42:00 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 20:42:00 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/10 20:42:00 No eventing-con process running
2021/12/10 20:42:00 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/10 20:42:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:42:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:42:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:42:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:42:13 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:42:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/10 20:42:14 Starting bucket ops to source bucket
2021/12/10 20:42:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 20:42:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:42:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:43:03 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 20:43:08 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 20:43:13 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 20:43:18 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/10 20:43:18 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/10 20:43:18 No eventing-con process running
2021/12/10 20:43:18 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/10 20:43:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:43:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 20:43:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 20:43:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:43:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:43:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (111.11s)
=== RUN   TestInterHandlerRecursion
2021/12/10 20:43:51 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 20:43:51 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 20:43:51 No eventing-con process running
2021/12/10 20:43:51 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/10 20:43:52 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:43:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:43:59 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:44:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:44:09 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 20:44:09 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 20:44:09 No eventing-con process running
2021/12/10 20:44:09 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/10 20:44:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:44:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:44:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:44:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:44:22 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:44:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/10 20:44:23 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/10 20:44:28 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 20:44:33 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 20:44:38 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 20:44:43 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 20:44:43 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/10 20:44:43 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:44:43 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/10 20:44:43 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/10 20:44:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:44:48 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 20:44:53 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 20:44:58 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 20:45:03 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 20:45:03 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 20:45:03 No eventing-con process running
2021/12/10 20:45:03 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/10 20:45:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:45:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:45:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:45:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:45:21 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 20:45:21 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 20:45:21 No eventing-con process running
2021/12/10 20:45:21 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/10 20:45:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:45:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:45:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:45:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.48s)
=== RUN   TestInterBucketRecursion
2021/12/10 20:45:39 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:45:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 20:45:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 20:45:40 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:45:40 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/10 20:45:45 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 20:45:50 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 20:45:55 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 20:46:00 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/10 20:46:00 App: TestInterBucketRecursion_function1 got deployed
2021/12/10 20:46:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:46:05 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 20:46:10 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 20:46:15 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 20:46:21 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/10 20:46:21 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/10 20:46:21 No eventing-con process running
2021/12/10 20:46:21 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/10 20:46:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:46:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:46:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:46:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:46:38 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/10 20:46:39 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/10 20:46:39 No eventing-con process running
2021/12/10 20:46:39 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/10 20:46:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:46:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:46:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:46:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.36s)
=== RUN   TestLargeHandler
2021/12/10 20:46:51 Changed force_compress value to false
2021/12/10 20:46:52 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:46:52 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/10 20:46:52 Changed force_compress value to true
2021/12/10 20:46:52 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:46:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/10 20:46:58 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 20:47:03 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 20:47:08 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 20:47:14 Waiting for app: TestLargeHandler_function to get deployed
2021/12/10 20:47:14 App: TestLargeHandler_function got deployed
2021/12/10 20:47:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:47:19 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 20:47:24 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 20:47:29 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 20:47:34 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/10 20:47:34 App: TestLargeHandler_function got un-deployed
2021/12/10 20:47:34 No eventing-con process running
2021/12/10 20:47:34 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/10 20:47:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:47:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:47:42 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:47:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.32s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/10 20:47:57 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 20:47:57 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 20:47:57 No eventing-con process running
2021/12/10 20:47:57 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/10 20:47:58 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:48:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:48:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:48:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:48:15 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 20:48:15 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 20:48:15 No eventing-con process running
2021/12/10 20:48:15 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/10 20:48:16 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:48:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:48:23 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:48:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:48:28 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:48:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/10 20:48:29 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/10 20:48:34 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 20:48:39 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 20:48:44 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 20:48:49 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/10 20:48:49 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/10 20:48:49 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:48:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/10 20:48:50 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/10 20:48:55 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 20:49:00 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 20:49:05 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 20:49:10 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/10 20:49:10 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/10 20:49:10 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/10 20:49:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:49:16 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 20:49:21 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 20:49:26 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 20:49:31 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/10 20:49:31 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/10 20:49:31 No eventing-con process running
2021/12/10 20:49:31 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/10 20:49:33 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:49:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:49:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:49:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:49:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:49:50 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 20:49:55 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 20:50:00 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 20:50:05 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/10 20:50:05 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/10 20:50:05 No eventing-con process running
2021/12/10 20:50:05 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/10 20:50:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:50:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:50:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:50:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.29s)
=== RUN   TestAllowInterBucketRecursion
2021/12/10 20:50:23 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:50:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 20:50:24 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 20:50:29 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 20:50:34 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 20:50:39 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 20:50:44 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/10 20:50:44 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/10 20:50:44 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:50:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/10 20:50:45 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/10 20:50:50 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 20:50:55 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 20:51:00 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 20:51:05 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/10 20:51:05 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/10 20:51:05 TestAllowInterBucketRecursion_function2 is deployed
2021/12/10 20:51:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:51:11 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 20:51:16 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 20:51:21 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 20:51:26 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/10 20:51:26 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/10 20:51:26 No eventing-con process running
2021/12/10 20:51:26 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/10 20:51:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:51:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:51:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:51:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:51:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:51:45 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 20:51:50 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 20:51:55 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 20:52:00 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/10 20:52:00 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/10 20:52:00 No eventing-con process running
2021/12/10 20:52:00 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/10 20:52:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:52:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:52:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:52:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (114.56s)
=== RUN   TestInterBucketRecursion2
2021/12/10 20:52:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:52:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:52:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:52:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:52:42 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:52:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 20:52:48 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 20:52:53 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 20:52:58 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 20:53:03 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/10 20:53:03 App: TestInterBucketRecursion2_function1 got deployed
2021/12/10 20:53:03 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:53:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/10 20:53:09 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 20:53:14 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 20:53:19 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 20:53:24 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/10 20:53:24 App: TestInterBucketRecursion2_function2 got deployed
2021/12/10 20:53:24 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:53:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/10 20:53:30 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 20:53:35 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 20:53:40 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 20:53:45 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/10 20:53:45 App: TestInterBucketRecursion2_function3 got deployed
2021/12/10 20:53:45 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:53:45 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/10 20:53:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:53:50 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 20:53:55 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 20:54:00 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 20:54:05 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/10 20:54:05 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/10 20:54:05 No eventing-con process running
2021/12/10 20:54:05 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/10 20:54:07 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:54:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:54:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:54:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:54:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:54:24 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 20:54:29 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 20:54:34 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 20:54:39 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/10 20:54:39 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/10 20:54:39 No eventing-con process running
2021/12/10 20:54:39 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/10 20:54:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:54:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:54:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:54:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:54:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 20:54:57 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 20:55:02 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 20:55:07 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 20:55:12 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/10 20:55:12 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/10 20:55:12 No eventing-con process running
2021/12/10 20:55:12 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/10 20:55:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:55:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:55:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:55:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:55:30 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/10 20:55:30 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/10 20:55:30 No eventing-con process running
2021/12/10 20:55:30 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/10 20:55:32 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:55:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:55:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:55:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (212.70s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/10 20:55:50 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 20:55:55 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/10 20:55:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 20:56:14 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 20:56:29 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 20:56:44 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 20:56:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 20:57:14 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 20:57:29 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 20:57:44 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/10 20:57:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/10 20:58:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:58:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 20:58:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 20:58:19 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/10 20:58:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:58:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 20:58:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 20:58:24 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/10 20:58:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:58:29 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/10 20:58:29 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/10 20:58:29 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:58:29 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/10 20:58:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:58:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:58:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:58:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:58:47 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 20:58:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 20:58:53 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 20:58:58 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 20:59:03 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 20:59:08 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/10 20:59:08 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: 247fa8fff8833e51f34d05319a3540a5
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 798fe84c93f474ef53f28c37b42bb067
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
Producer node: 127.0.0.1:9302	Node UUID: 5367ac45a03a04ae59b1cfdb9db613b1
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]


2021/12/10 20:59:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:59:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 20:59:14 Rebalance progress: 29.16666666666666
2021/12/10 20:59:19 Rebalance progress: 29.16666666666666
2021/12/10 20:59:24 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 5367ac45a03a04ae59b1cfdb9db613b1
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
Producer node: 172.31.2.134:9300	Node UUID: 247fa8fff8833e51f34d05319a3540a5
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 798fe84c93f474ef53f28c37b42bb067
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]


2021/12/10 20:59:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:59:24 handler_test.go:1674: Deleting source bucket: default
2021/12/10 20:59:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 20:59:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 20:59:31 Rebalance progress: 29.16666666666666
2021/12/10 20:59:36 Rebalance progress: 29.16666666666666
2021/12/10 20:59:41 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 906
	vb list dump: [0-5, 7-13, 15-17, 19-29, 31-33, 35-41, 43-47, 49-62, 64, 66-78, 80-83, 85, 87-107, 109-110, 112-119, 121-130, 132, 135-136, 138, 140-151, 153-160, 162-165, 167-170, 173-180, 182-205, 207-230, 232-236, 238-256, 258-260, 262-274, 277-282, 284-289, 291-292, 295-321, 323-327, 329-340, 342-345, 347-360, 362-366, 368-371, 373-380, 382-393, 395-417, 419-427, 429, 435-440, 442-446, 449-512, 514, 516, 518-525, 527-528, 530, 532-535, 537-559, 561-562, 565-567, 569-586, 588-595, 597-604, 606-607, 609-619, 621-629, 631-640, 642-661, 663-665, 667, 669-680, 684-692, 694-695, 697-704, 707, 709-710, 712-717, 719-725, 727-737, 739-775, 777-782, 784-792, 794-803, 805-807, 809, 812-814, 816, 818-845, 847-863, 865, 867-869, 871-883, 885-897, 899, 901-910, 912-913, 915, 917-935, 937-938, 940-944, 948-963, 965-966, 968-997, 999-1014, 1016-1020, 1022-1023]
	stream status: stopped
	len: 118
	vb list dump: [6, 14, 18, 30, 34, 42, 48, 63, 65, 79, 84, 86, 108, 111, 120, 131, 133-134, 137, 139, 152, 161, 166, 171-172, 181, 206, 231, 237, 257, 261, 275-276, 283, 290, 293-294, 322, 328, 341, 346, 361, 367, 372, 381, 394, 418, 428, 430-434, 441, 447-448, 513, 515, 517, 526, 529, 531, 536, 560, 563-564, 568, 587, 596, 605, 608, 620, 630, 641, 662, 666, 668, 681-683, 693, 696, 705-706, 708, 711, 718, 726, 738, 776, 783, 793, 804, 808, 810-811, 815, 817, 846, 864, 866, 870, 884, 898, 900, 911, 914, 916, 936, 939, 945-947, 964, 967, 998, 1015, 1021]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: 798fe84c93f474ef53f28c37b42bb067
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 98
	v: [342-345, 347-360, 362-366, 368-371, 373-380, 382-393, 395-417, 419-427, 429, 435-440, 442-446, 449-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 103
	v: [456-512, 514, 516, 518-525, 527-528, 530, 532-535, 537-559, 561-562, 565-567, 569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-586, 588-595, 597-604, 606-607, 609-619, 621-629, 631-640, 642-661, 663-665, 667, 669-680]
Producer node: 127.0.0.1:9302	Node UUID: 5367ac45a03a04ae59b1cfdb9db613b1
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-119, 121-130, 132, 135-136, 138, 140-151, 153-160, 162-165, 167-170, 173-180, 182-205, 207-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-230, 232-236, 238-256, 258-260, 262-274, 277-282, 284-289, 291-292, 295-321, 323-327, 329-340]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [0-5, 7-13, 15-17, 19-29, 31-33, 35-41, 43-47, 49-62, 64, 66-78, 80-83, 85, 87-107, 109-110, 112-113]
Producer node: 172.31.2.134:9300	Node UUID: 247fa8fff8833e51f34d05319a3540a5
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-803, 805-807, 809, 812-814, 816, 818-845, 847-863, 865, 867-869, 871-883, 885-897, 899, 901-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [912-913, 915, 917-935, 937-938, 940-944, 948-963, 965-966, 968-997, 999-1014, 1016-1020, 1022-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [684-692, 694-695, 697-704, 707, 709-710, 712-717, 719-725, 727-737, 739-775, 777-782, 784-792, 794-796]


2021/12/10 20:59:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 20:59:46 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/10 20:59:46 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/10 20:59:51 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/10 20:59:51 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/10 20:59:51 eventingSetup.go:46: No eventing-con process running
2021/12/10 20:59:51 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/10 20:59:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 20:59:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 20:59:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:00:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:00:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/10 21:00:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 21:00:10 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (267.06s)
=== RUN   TestN1QLRecursion
2021/12/10 21:00:12 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 21:00:13 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/10 21:00:13 Removing node(s): [] from the cluster
2021/12/10 21:00:18 Rebalance progress: 100
2021/12/10 21:00:24 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:00:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:00:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:00:40 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:00:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 21:00:42 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 21:00:47 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 21:00:52 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 21:00:57 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 21:01:02 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/10 21:01:02 App: TestN1QLRecursion_function1 got deployed
2021/12/10 21:01:02 TestN1QLRecursion_function1 is deployed
2021/12/10 21:01:02 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:01:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/10 21:01:03 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/10 21:01:08 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 21:01:13 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 21:01:18 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 21:01:23 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/10 21:01:23 App: TestN1QLRecursion_function3 got deployed
2021/12/10 21:01:23 TestN1QLRecursion_function3 is deployed
2021/12/10 21:01:23 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:01:23 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/10 21:01:23 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/10 21:01:23 Success: TestN1QLRecursion_function2 is not deployed
2021/12/10 21:01:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:01:28 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 21:01:33 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 21:01:38 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 21:01:43 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/10 21:01:43 App: TestN1QLRecursion_function1 got un-deployed
2021/12/10 21:01:43 No eventing-con process running
2021/12/10 21:01:43 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/10 21:01:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:01:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:01:52 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:01:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:01:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:02:02 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 21:02:07 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 21:02:12 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 21:02:17 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/10 21:02:17 App: TestN1QLRecursion_function3 got un-deployed
2021/12/10 21:02:17 No eventing-con process running
2021/12/10 21:02:17 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/10 21:02:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:02:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:02:26 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:02:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:02:32 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 21:02:37 Rebalance progress: 100
--- PASS: TestN1QLRecursion (144.98s)
=== RUN   TestN1QLAllowRecursion
2021/12/10 21:02:37 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 21:02:38 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/10 21:02:38 Removing node(s): [] from the cluster
2021/12/10 21:02:43 Rebalance progress: 100
2021/12/10 21:02:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:02:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:03:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:03:05 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:03:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/10 21:03:06 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/10 21:03:11 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 21:03:16 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 21:03:21 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 21:03:26 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/10 21:03:26 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/10 21:03:26 TestN1QLAllowRecursion_function1 is deployed
2021/12/10 21:03:26 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:03:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/10 21:03:27 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/10 21:03:32 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 21:03:37 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 21:03:42 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 21:03:47 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/10 21:03:47 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/10 21:03:47 TestN1QLAllowRecursion_function3 is deployed
2021/12/10 21:03:47 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:03:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/10 21:03:49 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/10 21:03:54 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 21:03:59 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 21:04:04 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 21:04:09 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/10 21:04:09 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/10 21:04:09 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/10 21:04:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:04:14 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 21:04:19 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 21:04:24 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 21:04:29 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/10 21:04:29 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/10 21:04:29 No eventing-con process running
2021/12/10 21:04:29 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/10 21:04:31 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:04:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:04:38 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:04:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:04:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:04:48 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 21:04:53 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 21:04:58 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 21:05:03 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/10 21:05:03 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/10 21:05:03 No eventing-con process running
2021/12/10 21:05:03 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/10 21:05:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:05:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:05:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:05:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:05:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:05:22 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 21:05:27 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 21:05:32 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 21:05:37 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/10 21:05:37 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/10 21:05:37 No eventing-con process running
2021/12/10 21:05:37 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/10 21:05:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:05:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:05:45 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:05:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:05:52 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 21:05:57 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (199.40s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/10 21:05:57 Sleeping for some time
2021/12/10 21:06:07 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 21:06:07 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/10 21:06:07 No eventing-con process running
2021/12/10 21:06:07 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/10 21:06:09 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:06:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:06:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:06:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:06:20 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:06:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 21:06:22 Starting bucket ops to source bucket
2021/12/10 21:07:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 21:07:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 21:07:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 21:07:16 Starting bucket ops to source bucket
2021/12/10 21:07:17 Starting bucket ops to source bucket
2021/12/10 21:08:23 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/10 21:08:28 Waiting for dst bucket item count to get to: 7000 curr count: 6300
2021/12/10 21:08:33 Waiting for dst bucket item count to get to: 7000 curr count: 6300
2021/12/10 21:08:33 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/10 21:08:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:08:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:08:38 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 21:08:43 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 21:08:48 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 21:08:53 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/10 21:08:53 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/10 21:08:53 No eventing-con process running
2021/12/10 21:08:53 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/10 21:08:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:09:05 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 21:09:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:09:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:09:16 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/10 21:09:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.64s)
=== RUN   TestTimerOverWriteSameReference
2021/12/10 21:09:21 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 21:09:21 App: TestTimerOverWriteSameReference got un-deployed
2021/12/10 21:09:21 No eventing-con process running
2021/12/10 21:09:21 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/10 21:09:23 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:09:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:09:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:09:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:09:35 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:09:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/10 21:09:41 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 21:09:46 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 21:09:51 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 21:09:56 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/10 21:09:56 App: TestTimerOverWriteSameReference got deployed
2021/12/10 21:10:06 Starting bucket ops to source bucket
2021/12/10 21:11:07 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/10 21:11:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:11:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:11:23 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 21:11:28 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 21:11:33 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 21:11:38 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/10 21:11:38 App: TestTimerOverWriteSameReference got un-deployed
2021/12/10 21:11:38 No eventing-con process running
2021/12/10 21:11:38 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/10 21:11:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:11:49 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 21:11:54 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 21:11:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:11:56 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:12:06 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/10 21:12:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.74s)
=== RUN   TestCancelTimerBucketop
2021/12/10 21:12:11 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 21:12:11 App: TestCancelTimerBucketop got un-deployed
2021/12/10 21:12:11 No eventing-con process running
2021/12/10 21:12:11 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/10 21:12:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:12:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:12:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:12:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:12:25 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:12:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/10 21:12:31 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 21:12:36 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 21:12:41 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 21:12:46 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/10 21:12:46 App: TestCancelTimerBucketop got deployed
2021/12/10 21:12:46 Starting bucket ops to source bucket
2021/12/10 21:12:53 Starting bucket ops to source bucket
2021/12/10 21:13:55 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 21:13:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:13:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:14:00 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 21:14:05 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 21:14:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 21:14:15 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/10 21:14:15 App: TestCancelTimerBucketop got un-deployed
2021/12/10 21:14:15 No eventing-con process running
2021/12/10 21:14:15 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/10 21:14:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:14:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 21:14:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:14:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:14:39 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 21:14:44 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 21:14:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (157.89s)
=== RUN   TestJSExpiryDate
2021/12/10 21:14:49 Starting bucket ops to source bucket
2021/12/10 21:15:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 21:15:02 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/10 21:15:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/10 21:15:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:15:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:15:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:15:38 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 21:15:43 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 21:15:48 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 21:15:53 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/10 21:15:53 App: TestJSExpiryDate got un-deployed
2021/12/10 21:15:53 No eventing-con process running
2021/12/10 21:15:54 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/10 21:15:55 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:16:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:16:03 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:16:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.58s)
=== RUN   TestBinaryDoc
2021/12/10 21:16:13 Starting bucket ops to source bucket
2021/12/10 21:16:13 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:16:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/10 21:16:14 Starting bucket ops to source bucket
2021/12/10 21:16:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:16:14 Deleting source bucket docs now
2021/12/10 21:16:14 Starting bucket ops to source bucket
2021/12/10 21:16:51 Waiting for dst bucket item count to get to: 32 curr count: 24
2021/12/10 21:16:56 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/10 21:16:56 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/10 21:16:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:16:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:17:01 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 21:17:06 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 21:17:11 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 21:17:16 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/10 21:17:16 App: TestBinaryDoc got un-deployed
2021/12/10 21:17:16 No eventing-con process running
2021/12/10 21:17:16 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/10 21:17:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:17:28 Waiting for dst bucket item count to get to: 0 curr count: 24
2021/12/10 21:17:33 Waiting for dst bucket item count to get to: 0 curr count: 6
2021/12/10 21:17:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:17:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:17:45 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/10 21:17:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.39s)
=== RUN   TestConstantBindings
2021/12/10 21:17:45 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:17:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/10 21:17:46 Starting bucket ops to source bucket
2021/12/10 21:17:52 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/10 21:17:57 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/10 21:18:02 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/10 21:18:07 Waiting for dst bucket item count to get to: 6000 curr count: 1895
2021/12/10 21:18:12 Waiting for dst bucket item count to get to: 6000 curr count: 1895
2021/12/10 21:18:12 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/10 21:18:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:18:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:18:17 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 21:18:22 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 21:18:27 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 21:18:32 Waiting for app: TestConstantBindings to get un-deployed
2021/12/10 21:18:32 App: TestConstantBindings got un-deployed
2021/12/10 21:18:32 No eventing-con process running
2021/12/10 21:18:32 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/10 21:18:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:18:44 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 21:18:49 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 21:18:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:18:51 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:19:01 Waiting for dst bucket item count to get to: 0 curr count: 5102
2021/12/10 21:19:06 Waiting for dst bucket item count to get to: 0 curr count: 5102
2021/12/10 21:19:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.51s)
=== RUN   TestCodeInsight
2021/12/10 21:19:11 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 21:19:11 App: TestCodeInsight got un-deployed
2021/12/10 21:19:11 No eventing-con process running
2021/12/10 21:19:11 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/10 21:19:12 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:19:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:19:19 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:19:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:19:24 Starting bucket ops to source bucket
2021/12/10 21:19:25 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:19:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/10 21:19:31 Waiting for app: TestCodeInsight to get deployed
2021/12/10 21:19:36 Waiting for app: TestCodeInsight to get deployed
2021/12/10 21:19:41 Waiting for app: TestCodeInsight to get deployed
2021/12/10 21:19:46 Waiting for app: TestCodeInsight to get deployed
2021/12/10 21:19:46 App: TestCodeInsight got deployed
2021/12/10 21:19:51 Waiting for dst bucket item count to get to: 1000 curr count: 465
2021/12/10 21:19:51 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 21:19:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:19:56 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 21:20:01 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 21:20:06 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 21:20:11 Waiting for app: TestCodeInsight to get un-deployed
2021/12/10 21:20:11 App: TestCodeInsight got un-deployed
2021/12/10 21:20:11 No eventing-con process running
2021/12/10 21:20:11 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/10 21:20:13 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:20:23 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 21:20:28 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 21:20:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:20:30 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:20:40 Waiting for dst bucket item count to get to: 0 curr count: 542
2021/12/10 21:20:45 Waiting for dst bucket item count to get to: 0 curr count: 542
2021/12/10 21:20:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.28s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.04s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12646.020s

Functional tests took 3 hours, 30 minutes and 49 seconds.

N1ql tests

2021/12/10 21:20:54 Started n1ql
2021/12/10 21:21:03 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/10 21:21:03 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/10 21:21:03 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 21:21:04 Removing node(s): [] from the cluster
2021/12/10 21:21:09 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/10 21:21:14 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/10 21:21:14 App: TestRecursiveMutationN1QL got un-deployed
2021/12/10 21:21:14 No eventing-con process running
2021/12/10 21:21:14 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/10 21:21:15 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:21:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:21:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:21:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:21:27 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:21:27 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.49s)
=== RUN   TestFlexReset1
2021/12/10 21:21:32 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 21:21:32 App: TestFlexReset1 got un-deployed
2021/12/10 21:21:32 No eventing-con process running
2021/12/10 21:21:32 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/10 21:21:34 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:21:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:21:41 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:21:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:21:46 Starting bucket ops to source bucket
2021/12/10 21:21:46 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:21:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/10 21:21:53 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 21:21:58 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 21:22:03 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 21:22:08 Waiting for app: TestFlexReset1 to get deployed
2021/12/10 21:22:08 App: TestFlexReset1 got deployed
2021/12/10 21:22:13 Waiting for dst bucket item count to get to: 1000 curr count: 41
2021/12/10 21:22:13 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 21:22:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:22:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:22:18 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 21:22:23 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 21:22:28 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 21:22:33 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/10 21:22:33 App: TestFlexReset1 got un-deployed
2021/12/10 21:22:33 No eventing-con process running
2021/12/10 21:22:33 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/10 21:22:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:22:45 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 21:22:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:22:46 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:22:57 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 21:22:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.24s)
=== RUN   TestFlexReset2
2021/12/10 21:23:02 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 21:23:02 App: TestFlexReset2 got un-deployed
2021/12/10 21:23:02 No eventing-con process running
2021/12/10 21:23:02 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/10 21:23:03 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:23:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:23:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:23:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:23:15 Starting bucket ops to source bucket
2021/12/10 21:23:16 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:23:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/10 21:23:22 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 21:23:27 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 21:23:32 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 21:23:37 Waiting for app: TestFlexReset2 to get deployed
2021/12/10 21:23:37 App: TestFlexReset2 got deployed
2021/12/10 21:23:42 Waiting for dst bucket item count to get to: 1000 curr count: 74
2021/12/10 21:23:42 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/10 21:23:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:23:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:23:47 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 21:23:52 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 21:23:57 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 21:24:02 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/10 21:24:02 App: TestFlexReset2 got un-deployed
2021/12/10 21:24:02 No eventing-con process running
2021/12/10 21:24:02 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/10 21:24:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:24:14 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/10 21:24:19 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 21:24:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:24:21 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:24:31 Waiting for dst bucket item count to get to: 0 curr count: 845
2021/12/10 21:24:36 Waiting for dst bucket item count to get to: 0 curr count: 845
2021/12/10 21:24:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.10s)
=== RUN   TestN1QLLabelledBreak
2021/12/10 21:24:41 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 21:24:41 App: TestN1QLLabelledBreak got un-deployed
2021/12/10 21:24:41 No eventing-con process running
2021/12/10 21:24:41 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/10 21:24:42 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:24:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:24:49 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:24:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:24:54 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:24:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/10 21:25:00 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 21:25:05 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 21:25:10 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 21:25:15 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/10 21:25:15 App: TestN1QLLabelledBreak got deployed
2021/12/10 21:25:15 Starting bucket ops to source bucket
2021/12/10 21:25:25 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/10 21:25:30 Waiting for dst bucket item count to get to: 10000 curr count: 382
2021/12/10 21:25:35 Waiting for dst bucket item count to get to: 10000 curr count: 382
2021/12/10 21:25:40 Waiting for dst bucket item count to get to: 10000 curr count: 4108
2021/12/10 21:25:45 Waiting for dst bucket item count to get to: 10000 curr count: 4108
2021/12/10 21:25:50 Waiting for dst bucket item count to get to: 10000 curr count: 8213
2021/12/10 21:25:55 Waiting for dst bucket item count to get to: 10000 curr count: 8213
2021/12/10 21:25:55 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 21:25:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:25:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:26:00 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 21:26:05 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 21:26:10 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 21:26:15 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/10 21:26:15 App: TestN1QLLabelledBreak got un-deployed
2021/12/10 21:26:15 No eventing-con process running
2021/12/10 21:26:15 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/10 21:26:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:26:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 21:26:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:26:29 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:26:39 Waiting for dst bucket item count to get to: 0 curr count: 8865
2021/12/10 21:26:44 Waiting for dst bucket item count to get to: 0 curr count: 3865
2021/12/10 21:26:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (127.97s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/10 21:26:49 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 21:26:49 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/10 21:26:49 No eventing-con process running
2021/12/10 21:26:49 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/10 21:26:50 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:26:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:26:57 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:27:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:27:02 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:27:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/10 21:27:08 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 21:27:13 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 21:27:18 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 21:27:23 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/10 21:27:23 App: TestN1QLUnlabelledBreak got deployed
2021/12/10 21:27:23 Starting bucket ops to source bucket
2021/12/10 21:27:33 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/10 21:27:38 Waiting for dst bucket item count to get to: 10000 curr count: 442
2021/12/10 21:27:43 Waiting for dst bucket item count to get to: 10000 curr count: 1419
2021/12/10 21:27:48 Waiting for dst bucket item count to get to: 10000 curr count: 4240
2021/12/10 21:27:53 Waiting for dst bucket item count to get to: 10000 curr count: 7124
2021/12/10 21:27:58 Waiting for dst bucket item count to get to: 10000 curr count: 8861
2021/12/10 21:27:58 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 21:27:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:27:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:28:03 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 21:28:08 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 21:28:13 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 21:28:18 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/10 21:28:18 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/10 21:28:18 No eventing-con process running
2021/12/10 21:28:18 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/10 21:28:20 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:28:30 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 21:28:35 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 21:28:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:28:36 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:28:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 21:28:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (122.84s)
=== RUN   TestN1QLThrowStatement
2021/12/10 21:28:51 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 21:28:51 App: TestN1QLThrowStatement got un-deployed
2021/12/10 21:28:51 No eventing-con process running
2021/12/10 21:28:51 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/10 21:28:53 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:28:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:29:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:29:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:29:05 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:29:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/10 21:29:11 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 21:29:16 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 21:29:21 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 21:29:26 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/10 21:29:26 App: TestN1QLThrowStatement got deployed
2021/12/10 21:29:26 Starting bucket ops to source bucket
2021/12/10 21:29:35 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/10 21:29:40 Waiting for dst bucket item count to get to: 10000 curr count: 255
2021/12/10 21:29:45 Waiting for dst bucket item count to get to: 10000 curr count: 255
2021/12/10 21:29:50 Waiting for dst bucket item count to get to: 10000 curr count: 3719
2021/12/10 21:29:55 Waiting for dst bucket item count to get to: 10000 curr count: 3719
2021/12/10 21:30:00 Waiting for dst bucket item count to get to: 10000 curr count: 7676
2021/12/10 21:30:05 Waiting for dst bucket item count to get to: 10000 curr count: 7676
2021/12/10 21:30:05 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/10 21:30:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:30:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:30:11 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 21:30:16 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 21:30:21 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 21:30:26 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/10 21:30:26 App: TestN1QLThrowStatement got un-deployed
2021/12/10 21:30:26 No eventing-con process running
2021/12/10 21:30:26 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/10 21:30:27 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:30:37 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/10 21:30:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:30:39 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:30:49 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/10 21:30:54 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 21:30:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (127.84s)
=== RUN   TestN1QLNestedForLoop
2021/12/10 21:30:59 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 21:30:59 App: TestN1QLNestedForLoop got un-deployed
2021/12/10 21:30:59 No eventing-con process running
2021/12/10 21:30:59 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/10 21:31:01 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:31:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:31:08 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:31:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:31:13 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:31:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/10 21:31:19 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 21:31:24 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 21:31:29 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 21:31:34 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/10 21:31:34 App: TestN1QLNestedForLoop got deployed
2021/12/10 21:31:34 Starting bucket ops to source bucket
2021/12/10 21:31:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 21:31:48 Waiting for dst bucket item count to get to: 5000 curr count: 111
2021/12/10 21:31:53 Waiting for dst bucket item count to get to: 5000 curr count: 379
2021/12/10 21:31:58 Waiting for dst bucket item count to get to: 5000 curr count: 1483
2021/12/10 21:32:03 Waiting for dst bucket item count to get to: 5000 curr count: 2683
2021/12/10 21:32:08 Waiting for dst bucket item count to get to: 5000 curr count: 3897
2021/12/10 21:32:13 Waiting for dst bucket item count to get to: 5000 curr count: 4929
2021/12/10 21:32:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 21:32:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:32:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:32:18 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 21:32:24 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 21:32:29 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 21:32:34 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/10 21:32:34 App: TestN1QLNestedForLoop got un-deployed
2021/12/10 21:32:34 No eventing-con process running
2021/12/10 21:32:34 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/10 21:32:35 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:32:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 21:32:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:32:47 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:32:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 21:32:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (122.97s)
=== RUN   TestN1QLAutoClose
2021/12/10 21:33:02 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 21:33:02 App: TestN1QLAutoClose got un-deployed
2021/12/10 21:33:02 No eventing-con process running
2021/12/10 21:33:02 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/10 21:33:04 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:33:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:33:11 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:33:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:33:16 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:33:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/10 21:33:22 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 21:33:27 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 21:33:32 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 21:33:37 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/10 21:33:37 App: TestN1QLAutoClose got deployed
2021/12/10 21:33:37 Starting bucket ops to source bucket
2021/12/10 21:33:43 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/10 21:33:48 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/10 21:33:53 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/10 21:33:58 Waiting for dst bucket item count to get to: 500 curr count: 23
2021/12/10 21:34:03 Waiting for dst bucket item count to get to: 500 curr count: 23
2021/12/10 21:34:08 Waiting for dst bucket item count to get to: 500 curr count: 45
2021/12/10 21:34:13 Waiting for dst bucket item count to get to: 500 curr count: 45
2021/12/10 21:34:18 Waiting for dst bucket item count to get to: 500 curr count: 67
2021/12/10 21:34:23 Waiting for dst bucket item count to get to: 500 curr count: 67
2021/12/10 21:34:28 Waiting for dst bucket item count to get to: 500 curr count: 90
2021/12/10 21:34:33 Waiting for dst bucket item count to get to: 500 curr count: 90
2021/12/10 21:34:38 Waiting for dst bucket item count to get to: 500 curr count: 110
2021/12/10 21:34:43 Waiting for dst bucket item count to get to: 500 curr count: 120
2021/12/10 21:34:48 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/10 21:34:53 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/10 21:34:58 Waiting for dst bucket item count to get to: 500 curr count: 142
2021/12/10 21:35:03 Waiting for dst bucket item count to get to: 500 curr count: 153
2021/12/10 21:35:08 Waiting for dst bucket item count to get to: 500 curr count: 164
2021/12/10 21:35:13 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/10 21:35:18 Waiting for dst bucket item count to get to: 500 curr count: 187
2021/12/10 21:35:23 Waiting for dst bucket item count to get to: 500 curr count: 198
2021/12/10 21:35:28 Waiting for dst bucket item count to get to: 500 curr count: 208
2021/12/10 21:35:33 Waiting for dst bucket item count to get to: 500 curr count: 219
2021/12/10 21:35:38 Waiting for dst bucket item count to get to: 500 curr count: 229
2021/12/10 21:35:43 Waiting for dst bucket item count to get to: 500 curr count: 239
2021/12/10 21:35:48 Waiting for dst bucket item count to get to: 500 curr count: 249
2021/12/10 21:35:53 Waiting for dst bucket item count to get to: 500 curr count: 261
2021/12/10 21:35:58 Waiting for dst bucket item count to get to: 500 curr count: 270
2021/12/10 21:36:03 Waiting for dst bucket item count to get to: 500 curr count: 282
2021/12/10 21:36:08 Waiting for dst bucket item count to get to: 500 curr count: 294
2021/12/10 21:36:13 Waiting for dst bucket item count to get to: 500 curr count: 301
2021/12/10 21:36:18 Waiting for dst bucket item count to get to: 500 curr count: 313
2021/12/10 21:36:23 Waiting for dst bucket item count to get to: 500 curr count: 322
2021/12/10 21:36:28 Waiting for dst bucket item count to get to: 500 curr count: 345
2021/12/10 21:36:33 Waiting for dst bucket item count to get to: 500 curr count: 345
2021/12/10 21:36:38 Waiting for dst bucket item count to get to: 500 curr count: 364
2021/12/10 21:36:43 Waiting for dst bucket item count to get to: 500 curr count: 364
2021/12/10 21:36:48 Waiting for dst bucket item count to get to: 500 curr count: 385
2021/12/10 21:36:53 Waiting for dst bucket item count to get to: 500 curr count: 385
2021/12/10 21:36:58 Waiting for dst bucket item count to get to: 500 curr count: 406
2021/12/10 21:37:03 Waiting for dst bucket item count to get to: 500 curr count: 406
2021/12/10 21:37:08 Waiting for dst bucket item count to get to: 500 curr count: 428
2021/12/10 21:37:13 Waiting for dst bucket item count to get to: 500 curr count: 428
2021/12/10 21:37:18 Waiting for dst bucket item count to get to: 500 curr count: 450
2021/12/10 21:37:23 Waiting for dst bucket item count to get to: 500 curr count: 450
2021/12/10 21:37:28 Waiting for dst bucket item count to get to: 500 curr count: 472
2021/12/10 21:37:33 Waiting for dst bucket item count to get to: 500 curr count: 472
2021/12/10 21:37:38 Waiting for dst bucket item count to get to: 500 curr count: 492
2021/12/10 21:37:43 Waiting for dst bucket item count to get to: 500 curr count: 492
2021/12/10 21:37:43 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/10 21:37:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:37:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:37:48 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 21:37:53 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 21:37:58 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 21:38:03 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/10 21:38:03 App: TestN1QLAutoClose got un-deployed
2021/12/10 21:38:03 No eventing-con process running
2021/12/10 21:38:03 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/10 21:38:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:38:15 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 21:38:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:38:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:38:27 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/10 21:38:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.46s)
=== RUN   TestN1QLPosParams
2021/12/10 21:38:32 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 21:38:32 App: TestN1QLPosParams got un-deployed
2021/12/10 21:38:32 No eventing-con process running
2021/12/10 21:38:32 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/10 21:38:33 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:38:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:38:40 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:38:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:38:45 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:38:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/10 21:38:51 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 21:38:56 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 21:39:01 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 21:39:06 Waiting for app: TestN1QLPosParams to get deployed
2021/12/10 21:39:06 App: TestN1QLPosParams got deployed
2021/12/10 21:39:06 Starting bucket ops to source bucket
2021/12/10 21:39:16 Waiting for dst bucket item count to get to: 5000 curr count: 596
2021/12/10 21:39:21 Waiting for dst bucket item count to get to: 5000 curr count: 596
2021/12/10 21:39:26 Waiting for dst bucket item count to get to: 5000 curr count: 596
2021/12/10 21:39:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 21:39:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:39:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:39:31 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 21:39:36 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 21:39:41 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 21:39:46 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/10 21:39:46 App: TestN1QLPosParams got un-deployed
2021/12/10 21:39:46 No eventing-con process running
2021/12/10 21:39:46 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/10 21:39:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:39:58 Waiting for dst bucket item count to get to: 0 curr count: 3836
2021/12/10 21:40:03 Waiting for dst bucket item count to get to: 0 curr count: 1336
2021/12/10 21:40:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:40:05 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:40:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 21:40:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.19s)
=== RUN   TestN1QLExhaustConnPool
2021/12/10 21:40:20 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 21:40:20 App: TestN1QLExhaustConnPool got un-deployed
2021/12/10 21:40:20 No eventing-con process running
2021/12/10 21:40:20 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/10 21:40:22 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:40:28 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:40:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:40:33 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:40:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/10 21:40:40 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 21:40:45 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 21:40:50 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 21:40:55 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/10 21:40:55 App: TestN1QLExhaustConnPool got deployed
2021/12/10 21:40:55 Starting bucket ops to source bucket
2021/12/10 21:41:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/10 21:41:05 Waiting for dst bucket item count to get to: 100 curr count: 40
2021/12/10 21:41:10 Waiting for dst bucket item count to get to: 100 curr count: 90
2021/12/10 21:41:10 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/10 21:41:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:41:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:41:15 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 21:41:20 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 21:41:25 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 21:41:30 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/10 21:41:30 App: TestN1QLExhaustConnPool got un-deployed
2021/12/10 21:41:30 No eventing-con process running
2021/12/10 21:41:30 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/10 21:41:32 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:41:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 21:41:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/10 21:41:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:41:48 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:41:58 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/10 21:42:03 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/10 21:42:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.58s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/10 21:42:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 21:42:08 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/10 21:42:08 No eventing-con process running
2021/12/10 21:42:08 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/10 21:42:10 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:42:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:42:17 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:42:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:42:22 Starting bucket ops to source bucket
2021/12/10 21:42:39 Starting bucket ops to source bucket
2021/12/10 21:42:57 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 15653
2021/12/10 21:42:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/10 21:43:03 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 21:43:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 21:43:13 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 21:43:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/10 21:43:18 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/10 21:43:18 Starting bucket ops to source bucket
2021/12/10 21:43:23 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 21:43:28 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 21:43:33 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 21:43:38 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 21:43:43 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 21:43:48 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 21:43:53 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 21:43:58 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 21:44:03 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/10 21:44:03 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/10 21:44:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:44:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:44:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 21:44:13 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 21:44:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 21:44:23 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/10 21:44:23 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/10 21:44:23 No eventing-con process running
2021/12/10 21:44:23 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/10 21:44:25 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:44:35 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/10 21:44:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:44:37 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:44:47 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/10 21:44:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (163.36s)
=== RUN   TestN1QLQueryConsistency
2021/12/10 21:44:52 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 21:44:52 App: n1qlquery_consistency got un-deployed
2021/12/10 21:44:52 No eventing-con process running
2021/12/10 21:44:52 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/10 21:44:54 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:44:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:45:00 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:45:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:45:05 Starting bucket ops to source bucket
2021/12/10 21:45:08 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:45:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/10 21:45:14 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 21:45:19 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 21:45:24 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 21:45:29 Waiting for app: n1qlquery_consistency to get deployed
2021/12/10 21:45:29 App: n1qlquery_consistency got deployed
2021/12/10 21:45:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/10 21:45:39 Waiting for dst bucket item count to get to: 5000 curr count: 4170
2021/12/10 21:45:44 Waiting for dst bucket item count to get to: 5000 curr count: 4170
2021/12/10 21:45:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/10 21:45:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/10 21:45:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:45:50 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 21:45:55 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 21:46:00 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 21:46:05 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/10 21:46:05 App: n1qlquery_consistency got un-deployed
2021/12/10 21:46:05 No eventing-con process running
2021/12/10 21:46:05 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/10 21:46:06 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:46:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/10 21:46:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:46:18 Context: Bucket flush, response code: 200 dump: 
2021/12/10 21:46:28 Waiting for dst bucket item count to get to: 0 curr count: 3924
2021/12/10 21:46:33 Waiting for dst bucket item count to get to: 0 curr count: 1424
2021/12/10 21:46:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.38s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1535.412s

N1ql tests took 0 hours, 25 minutes and 39 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/10 21:48:08 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:47898->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:47898->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/10 21:48:27 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 21:48:27 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/10 21:48:27 No eventing-con process running
2021/12/10 21:48:27 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/10 21:48:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:48:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:48:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:48:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:48:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:48:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:48:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:48:45 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:48:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 21:48:51 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 21:48:56 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 21:49:01 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 21:49:06 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/10 21:49:06 App: TestDuplicateWithEventingRebIn got deployed
2021/12/10 21:49:06 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 21:49:06 Starting bucket ops to source bucket
2021/12/10 21:49:20 Removing node(s): [] from the cluster
2021/12/10 21:49:25 Rebalance progress: 0
2021/12/10 21:49:30 Rebalance progress: 0
2021/12/10 21:49:35 Rebalance progress: 25
2021/12/10 21:49:40 Rebalance progress: 25
2021/12/10 21:49:45 Rebalance progress: 25
2021/12/10 21:49:50 Rebalance progress: 98.91304347826086
2021/12/10 21:49:55 Rebalance progress: 98.91304347826086
2021/12/10 21:50:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:50:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:50:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 21:50:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 21:50:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 21:50:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 21:50:06 Rebalance progress: 25
2021/12/10 21:50:11 Rebalance progress: 25
2021/12/10 21:50:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:50:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:50:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 21:50:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 21:50:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 21:50:16 Waiting for src bucket item count to get to: 200000 curr count: 104049
2021/12/10 21:50:21 Waiting for src bucket item count to get to: 200000 curr count: 104049
2021/12/10 21:50:26 Waiting for src bucket item count to get to: 200000 curr count: 120845
2021/12/10 21:50:31 Waiting for src bucket item count to get to: 200000 curr count: 120845
2021/12/10 21:50:36 Waiting for src bucket item count to get to: 200000 curr count: 137738
2021/12/10 21:50:41 Waiting for src bucket item count to get to: 200000 curr count: 137738
2021/12/10 21:50:46 Waiting for src bucket item count to get to: 200000 curr count: 154697
2021/12/10 21:50:51 Waiting for src bucket item count to get to: 200000 curr count: 154697
2021/12/10 21:50:56 Waiting for src bucket item count to get to: 200000 curr count: 171489
2021/12/10 21:51:01 Waiting for src bucket item count to get to: 200000 curr count: 171489
2021/12/10 21:51:06 Waiting for src bucket item count to get to: 200000 curr count: 188430
2021/12/10 21:51:11 Waiting for src bucket item count to get to: 200000 curr count: 188430
2021/12/10 21:51:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 21:51:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 21:51:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:51:21 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 21:51:26 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 21:51:31 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 21:51:36 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/10 21:51:36 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/10 21:51:36 No eventing-con process running
2021/12/10 21:51:36 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/10 21:51:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:51:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:51:47 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 21:51:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:51:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:51:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:51:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 21:51:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (222.42s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/10 21:52:04 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 21:52:05 Removing node(s): [] from the cluster
2021/12/10 21:52:10 Rebalance progress: 100
2021/12/10 21:52:15 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 21:52:15 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/10 21:52:15 No eventing-con process running
2021/12/10 21:52:15 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/10 21:52:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:52:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:52:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:52:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:52:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:52:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:52:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:52:33 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:52:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 21:52:39 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 21:52:44 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 21:52:49 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 21:52:54 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/10 21:52:54 App: TestDuplicateWithEventingRebOut got deployed
2021/12/10 21:52:54 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 21:52:54 Starting bucket ops to source bucket
2021/12/10 21:52:59 Rebalance progress: 25
2021/12/10 21:53:04 Rebalance progress: 25
2021/12/10 21:53:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:53:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:53:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 21:53:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 21:53:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 21:53:09 Waiting for src bucket item count to get to: 200000 curr count: 8146
2021/12/10 21:53:14 Waiting for src bucket item count to get to: 200000 curr count: 24943
2021/12/10 21:53:19 Waiting for src bucket item count to get to: 200000 curr count: 24943
2021/12/10 21:53:24 Waiting for src bucket item count to get to: 200000 curr count: 41581
2021/12/10 21:53:29 Waiting for src bucket item count to get to: 200000 curr count: 41581
2021/12/10 21:53:34 Waiting for src bucket item count to get to: 200000 curr count: 58139
2021/12/10 21:53:39 Waiting for src bucket item count to get to: 200000 curr count: 58139
2021/12/10 21:53:44 Waiting for src bucket item count to get to: 200000 curr count: 58139
2021/12/10 21:53:49 Waiting for src bucket item count to get to: 200000 curr count: 74794
2021/12/10 21:53:54 Waiting for src bucket item count to get to: 200000 curr count: 74794
2021/12/10 21:53:59 Waiting for src bucket item count to get to: 200000 curr count: 90591
2021/12/10 21:54:04 Waiting for src bucket item count to get to: 200000 curr count: 107569
2021/12/10 21:54:09 Waiting for src bucket item count to get to: 200000 curr count: 107569
2021/12/10 21:54:14 Waiting for src bucket item count to get to: 200000 curr count: 124311
2021/12/10 21:54:19 Waiting for src bucket item count to get to: 200000 curr count: 124311
2021/12/10 21:54:24 Waiting for src bucket item count to get to: 200000 curr count: 140743
2021/12/10 21:54:29 Waiting for src bucket item count to get to: 200000 curr count: 140743
2021/12/10 21:54:34 Waiting for src bucket item count to get to: 200000 curr count: 157701
2021/12/10 21:54:39 Waiting for src bucket item count to get to: 200000 curr count: 157701
2021/12/10 21:54:44 Waiting for src bucket item count to get to: 200000 curr count: 174535
2021/12/10 21:54:49 Waiting for src bucket item count to get to: 200000 curr count: 174535
2021/12/10 21:54:54 Waiting for src bucket item count to get to: 200000 curr count: 190225
2021/12/10 21:54:59 Waiting for src bucket item count to get to: 200000 curr count: 190225
2021/12/10 21:55:04 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 21:55:04 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 21:55:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:55:09 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 21:55:14 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 21:55:19 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 21:55:24 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/10 21:55:24 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/10 21:55:24 No eventing-con process running
2021/12/10 21:55:25 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/10 21:55:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:55:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:55:36 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 21:55:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:55:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:55:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:55:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 21:55:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (228.76s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/10 21:55:58 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 21:55:58 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/10 21:55:58 No eventing-con process running
2021/12/10 21:55:58 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/10 21:56:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:56:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:56:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:56:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:56:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:56:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:56:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:56:16 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 21:56:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 21:56:22 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 21:56:27 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 21:56:32 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 21:56:37 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/10 21:56:37 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/10 21:56:37 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 21:56:37 Starting bucket ops to source bucket
2021/12/10 21:56:39 Removing node(s): [] from the cluster
2021/12/10 21:56:44 Rebalance progress: 25
2021/12/10 21:56:49 Rebalance progress: 25
2021/12/10 21:56:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:56:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:56:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 21:56:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/10 21:56:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 21:56:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 21:57:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 21:57:06 Rebalance progress: 16.66666666666667
2021/12/10 21:57:11 Rebalance progress: 16.66666666666667
2021/12/10 21:57:16 Rebalance progress: 16.66666666666667
2021/12/10 21:57:21 Rebalance progress: 63.16425120772948
2021/12/10 21:57:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:57:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:57:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 21:57:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 21:57:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 21:57:31 Rebalance progress: 16.66666666666667
2021/12/10 21:57:36 Rebalance progress: 16.66666666666667
2021/12/10 21:57:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:57:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:57:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 21:57:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 21:57:41 Waiting for src bucket item count to get to: 200000 curr count: 78293
2021/12/10 21:57:46 Waiting for src bucket item count to get to: 200000 curr count: 94887
2021/12/10 21:57:51 Waiting for src bucket item count to get to: 200000 curr count: 94887
2021/12/10 21:57:56 Waiting for src bucket item count to get to: 200000 curr count: 111089
2021/12/10 21:58:01 Waiting for src bucket item count to get to: 200000 curr count: 111089
2021/12/10 21:58:06 Waiting for src bucket item count to get to: 200000 curr count: 127497
2021/12/10 21:58:11 Waiting for src bucket item count to get to: 200000 curr count: 127497
2021/12/10 21:58:16 Waiting for src bucket item count to get to: 200000 curr count: 143894
2021/12/10 21:58:21 Waiting for src bucket item count to get to: 200000 curr count: 143894
2021/12/10 21:58:26 Waiting for src bucket item count to get to: 200000 curr count: 160549
2021/12/10 21:58:31 Waiting for src bucket item count to get to: 200000 curr count: 160549
2021/12/10 21:58:36 Waiting for src bucket item count to get to: 200000 curr count: 176414
2021/12/10 21:58:41 Waiting for src bucket item count to get to: 200000 curr count: 176414
2021/12/10 21:58:46 Waiting for src bucket item count to get to: 200000 curr count: 193175
2021/12/10 21:58:51 Waiting for src bucket item count to get to: 200000 curr count: 193175
2021/12/10 21:58:56 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 21:58:56 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 21:58:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 21:59:01 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 21:59:06 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 21:59:11 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 21:59:16 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/10 21:59:16 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/10 21:59:16 No eventing-con process running
2021/12/10 21:59:16 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/10 21:59:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:59:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:59:28 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 21:59:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:59:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:59:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:59:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (226.85s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/10 21:59:45 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 21:59:45 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/10 21:59:45 No eventing-con process running
2021/12/10 21:59:45 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/10 21:59:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:59:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:59:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 21:59:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 21:59:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 21:59:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:00:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:00:03 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:00:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 22:00:09 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 22:00:14 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 22:00:19 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 22:00:24 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/10 22:00:24 App: TestDuplicateWithKVRebIn got deployed
2021/12/10 22:00:24 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 22:00:24 Starting bucket ops to source bucket
2021/12/10 22:00:25 Removing node(s): [] from the cluster
2021/12/10 22:00:30 Rebalance progress: 16.66666666666667
2021/12/10 22:00:35 Rebalance progress: 16.66666666666667
2021/12/10 22:00:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:00:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:00:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:00:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 22:00:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:00:45 Rebalance progress: 16.66666666666667
2021/12/10 22:00:50 Rebalance progress: 16.66666666666667
2021/12/10 22:00:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:00:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:00:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:00:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 22:00:55 Waiting for src bucket item count to get to: 200000 curr count: 39575
2021/12/10 22:01:00 Waiting for src bucket item count to get to: 200000 curr count: 39575
2021/12/10 22:01:05 Waiting for src bucket item count to get to: 200000 curr count: 56107
2021/12/10 22:01:10 Waiting for src bucket item count to get to: 200000 curr count: 56107
2021/12/10 22:01:15 Waiting for src bucket item count to get to: 200000 curr count: 72749
2021/12/10 22:01:20 Waiting for src bucket item count to get to: 200000 curr count: 88429
2021/12/10 22:01:25 Waiting for src bucket item count to get to: 200000 curr count: 88429
2021/12/10 22:01:30 Waiting for src bucket item count to get to: 200000 curr count: 105248
2021/12/10 22:01:35 Waiting for src bucket item count to get to: 200000 curr count: 105248
2021/12/10 22:01:40 Waiting for src bucket item count to get to: 200000 curr count: 121820
2021/12/10 22:01:45 Waiting for src bucket item count to get to: 200000 curr count: 121820
2021/12/10 22:01:50 Waiting for src bucket item count to get to: 200000 curr count: 138563
2021/12/10 22:01:55 Waiting for src bucket item count to get to: 200000 curr count: 138563
2021/12/10 22:02:00 Waiting for src bucket item count to get to: 200000 curr count: 138563
2021/12/10 22:02:05 Waiting for src bucket item count to get to: 200000 curr count: 155219
2021/12/10 22:02:10 Waiting for src bucket item count to get to: 200000 curr count: 155219
2021/12/10 22:02:15 Waiting for src bucket item count to get to: 200000 curr count: 171896
2021/12/10 22:02:20 Waiting for src bucket item count to get to: 200000 curr count: 171896
2021/12/10 22:02:25 Waiting for src bucket item count to get to: 200000 curr count: 187425
2021/12/10 22:02:30 Waiting for src bucket item count to get to: 200000 curr count: 187425
2021/12/10 22:02:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 22:02:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 22:02:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:02:40 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 22:02:45 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 22:02:50 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 22:02:55 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/10 22:02:55 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/10 22:02:55 No eventing-con process running
2021/12/10 22:02:55 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/10 22:02:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:03:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:03:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 22:03:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:03:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:03:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:03:19 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 22:03:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (224.04s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/10 22:03:24 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 22:03:24 Removing node(s): [] from the cluster
2021/12/10 22:03:29 Rebalance progress: 100
2021/12/10 22:03:34 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 22:03:34 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/10 22:03:34 No eventing-con process running
2021/12/10 22:03:34 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/10 22:03:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:03:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:03:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:03:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:03:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:03:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:03:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:03:53 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:03:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 22:03:59 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 22:04:04 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 22:04:09 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 22:04:14 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/10 22:04:14 App: TestDuplicateWithKVRebOut got deployed
2021/12/10 22:04:14 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:04:14 Starting bucket ops to source bucket
2021/12/10 22:04:19 Rebalance progress: 16.66666666666667
2021/12/10 22:04:24 Rebalance progress: 16.66666666666667
2021/12/10 22:04:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:04:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:04:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:04:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 22:04:29 Waiting for src bucket item count to get to: 200000 curr count: 7807
2021/12/10 22:04:34 Waiting for src bucket item count to get to: 200000 curr count: 23915
2021/12/10 22:04:39 Waiting for src bucket item count to get to: 200000 curr count: 23915
2021/12/10 22:04:44 Waiting for src bucket item count to get to: 200000 curr count: 40171
2021/12/10 22:04:49 Waiting for src bucket item count to get to: 200000 curr count: 40171
2021/12/10 22:04:54 Waiting for src bucket item count to get to: 200000 curr count: 56962
2021/12/10 22:04:59 Waiting for src bucket item count to get to: 200000 curr count: 56962
2021/12/10 22:05:04 Waiting for src bucket item count to get to: 200000 curr count: 56962
2021/12/10 22:05:09 Waiting for src bucket item count to get to: 200000 curr count: 73745
2021/12/10 22:05:14 Waiting for src bucket item count to get to: 200000 curr count: 73745
2021/12/10 22:05:19 Waiting for src bucket item count to get to: 200000 curr count: 89414
2021/12/10 22:05:24 Waiting for src bucket item count to get to: 200000 curr count: 89414
2021/12/10 22:05:29 Waiting for src bucket item count to get to: 200000 curr count: 106144
2021/12/10 22:05:34 Waiting for src bucket item count to get to: 200000 curr count: 123023
2021/12/10 22:05:39 Waiting for src bucket item count to get to: 200000 curr count: 123023
2021/12/10 22:05:44 Waiting for src bucket item count to get to: 200000 curr count: 139831
2021/12/10 22:05:49 Waiting for src bucket item count to get to: 200000 curr count: 139831
2021/12/10 22:05:54 Waiting for src bucket item count to get to: 200000 curr count: 156683
2021/12/10 22:05:59 Waiting for src bucket item count to get to: 200000 curr count: 156683
2021/12/10 22:06:04 Waiting for src bucket item count to get to: 200000 curr count: 173478
2021/12/10 22:06:09 Waiting for src bucket item count to get to: 200000 curr count: 173478
2021/12/10 22:06:14 Waiting for src bucket item count to get to: 200000 curr count: 189053
2021/12/10 22:06:19 Waiting for src bucket item count to get to: 200000 curr count: 189053
2021/12/10 22:06:24 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 22:06:24 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 22:06:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:06:29 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 22:06:34 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 22:06:39 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 22:06:44 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/10 22:06:44 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/10 22:06:44 No eventing-con process running
2021/12/10 22:06:44 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/10 22:06:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:06:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:06:56 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 22:06:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:06:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:07:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:07:08 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 22:07:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (229.09s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/10 22:07:18 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 22:07:18 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/10 22:07:18 No eventing-con process running
2021/12/10 22:07:18 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/10 22:07:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:07:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:07:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:07:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:07:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:07:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:07:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:07:36 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:07:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 22:07:43 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 22:07:48 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 22:07:53 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 22:07:58 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/10 22:07:58 App: TestDuplicateWithKVSwapReb got deployed
2021/12/10 22:07:58 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 22:07:58 Starting bucket ops to source bucket
2021/12/10 22:07:58 Removing node(s): [] from the cluster
2021/12/10 22:08:03 Rebalance progress: 16.66666666666667
2021/12/10 22:08:08 Rebalance progress: 16.66666666666667
2021/12/10 22:08:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:08:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:08:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:08:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 22:08:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 22:08:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:08:18 Rebalance progress: 16.66666666666667
2021/12/10 22:08:23 Rebalance progress: 16.66666666666667
2021/12/10 22:08:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:08:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:08:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:08:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 22:08:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 22:08:33 Rebalance progress: 16.66666666666667
2021/12/10 22:08:38 Rebalance progress: 16.66666666666667
2021/12/10 22:08:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:08:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:08:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:08:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 22:08:43 Waiting for src bucket item count to get to: 200000 curr count: 66877
2021/12/10 22:08:48 Waiting for src bucket item count to get to: 200000 curr count: 66877
2021/12/10 22:08:53 Waiting for src bucket item count to get to: 200000 curr count: 82551
2021/12/10 22:08:58 Waiting for src bucket item count to get to: 200000 curr count: 82551
2021/12/10 22:09:03 Waiting for src bucket item count to get to: 200000 curr count: 99474
2021/12/10 22:09:08 Waiting for src bucket item count to get to: 200000 curr count: 99474
2021/12/10 22:09:13 Waiting for src bucket item count to get to: 200000 curr count: 116311
2021/12/10 22:09:18 Waiting for src bucket item count to get to: 200000 curr count: 116311
2021/12/10 22:09:23 Waiting for src bucket item count to get to: 200000 curr count: 116311
2021/12/10 22:09:28 Waiting for src bucket item count to get to: 200000 curr count: 132768
2021/12/10 22:09:33 Waiting for src bucket item count to get to: 200000 curr count: 132768
2021/12/10 22:09:38 Waiting for src bucket item count to get to: 200000 curr count: 149202
2021/12/10 22:09:43 Waiting for src bucket item count to get to: 200000 curr count: 149202
2021/12/10 22:09:48 Waiting for src bucket item count to get to: 200000 curr count: 165749
2021/12/10 22:09:53 Waiting for src bucket item count to get to: 200000 curr count: 165749
2021/12/10 22:09:58 Waiting for src bucket item count to get to: 200000 curr count: 181691
2021/12/10 22:10:03 Waiting for src bucket item count to get to: 200000 curr count: 181691
2021/12/10 22:10:08 Waiting for src bucket item count to get to: 200000 curr count: 198345
2021/12/10 22:10:13 Waiting for src bucket item count to get to: 200000 curr count: 198345
2021/12/10 22:10:18 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 22:10:18 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 22:10:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:10:23 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 22:10:28 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 22:10:33 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 22:10:38 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/10 22:10:38 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/10 22:10:38 No eventing-con process running
2021/12/10 22:10:39 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/10 22:10:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:10:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:10:50 Waiting for dst bucket item count to get to: 0 curr count: 65651
2021/12/10 22:10:55 Waiting for dst bucket item count to get to: 0 curr count: 65651
2021/12/10 22:10:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:10:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:11:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:11:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/10 22:11:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (239.08s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/10 22:11:17 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:11:17 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 22:11:17 No eventing-con process running
2021/12/10 22:11:17 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/10 22:11:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:11:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:11:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:11:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:11:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:11:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:11:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:11:35 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:11:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/10 22:11:42 Starting bucket ops to source bucket
2021/12/10 22:11:47 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 22:11:52 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 22:11:57 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 22:11:57 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:11:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:11:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:11:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 22:11:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:11:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:12:02 Rebalance progress: 16.66666666666667
2021/12/10 22:12:07 Rebalance progress: 16.66666666666667
2021/12/10 22:12:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:12:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:12:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:12:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 22:12:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:12:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:12:18 Rebalance progress: 16.66666666666667
2021/12/10 22:12:23 Rebalance progress: 16.66666666666667
2021/12/10 22:12:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:12:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:12:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:12:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/10 22:12:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 22:12:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:12:38 Rebalance progress: 12.5
2021/12/10 22:12:43 Rebalance progress: 12.5
2021/12/10 22:12:48 Rebalance progress: 12.5
2021/12/10 22:12:53 Rebalance progress: 98.62204724409449
2021/12/10 22:12:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:12:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:12:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:12:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:13:03 Rebalance progress: 12.5
2021/12/10 22:13:08 Rebalance progress: 12.5
2021/12/10 22:13:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:13:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:13:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:13:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 22:13:18 Rebalance progress: 12.5
2021/12/10 22:13:23 Rebalance progress: 12.5
2021/12/10 22:13:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:13:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:13:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:13:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 22:13:33 Rebalance progress: 12.5
2021/12/10 22:13:38 Rebalance progress: 12.5
2021/12/10 22:13:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:13:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:13:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:13:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:13:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:13:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:13:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:14:03 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:14:03 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 22:14:03 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:14:03 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/10 22:14:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:14:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:14:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12158
2021/12/10 22:14:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:14:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:14:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:14:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12158
2021/12/10 22:14:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (199.93s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/10 22:14:37 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 22:14:37 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/10 22:14:37 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:14:37 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/10 22:14:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:14:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:14:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:14:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:14:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:14:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:14:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:14:55 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:14:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/10 22:15:02 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 22:15:07 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/10 22:15:12 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/10 22:15:17 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/10 22:15:17 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:15:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:15:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:15:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:15:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:15:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 22:15:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:15:23 Rebalance progress: 12.5
2021/12/10 22:15:28 Rebalance progress: 12.5
2021/12/10 22:15:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:15:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:15:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:15:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 22:15:38 Rebalance progress: 12.5
2021/12/10 22:15:43 Rebalance progress: 12.5
2021/12/10 22:15:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:15:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:15:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:15:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:15:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 22:15:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 22:16:03 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 22:16:08 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/10 22:16:08 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/10 22:16:08 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:16:08 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/10 22:16:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:16:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:16:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 467
2021/12/10 22:16:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 467
2021/12/10 22:16:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:16:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:16:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:16:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4631
2021/12/10 22:16:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.81s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/10 22:16:47 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 22:16:47 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 22:16:47 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:16:47 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/10 22:16:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:16:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:16:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:16:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:17:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:17:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:17:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:17:05 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:17:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/10 22:17:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 22:17:16 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/10 22:17:21 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/10 22:17:26 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/10 22:17:26 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:17:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:17:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:17:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:17:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:17:32 Rebalance progress: 12.5
2021/12/10 22:17:37 Rebalance progress: 12.5
2021/12/10 22:17:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:17:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:17:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:17:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:17:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:17:47 Rebalance progress: 12.5
2021/12/10 22:17:52 Rebalance progress: 12.5
2021/12/10 22:17:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:17:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:17:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:17:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 22:17:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:18:03 Rebalance progress: 12.5
2021/12/10 22:18:08 Rebalance progress: 12.5
2021/12/10 22:18:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:18:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:18:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:18:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:18:18 Rebalance progress: 12.5
2021/12/10 22:18:23 Rebalance progress: 12.5
2021/12/10 22:18:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:18:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:18:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:18:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 22:18:33 Rebalance progress: 12.5
2021/12/10 22:18:38 Rebalance progress: 12.5
2021/12/10 22:18:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:18:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:18:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:18:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 22:18:48 Rebalance progress: 12.5
2021/12/10 22:18:53 Rebalance progress: 12.5
2021/12/10 22:18:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:18:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:18:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:18:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:19:03 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 22:19:08 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 22:19:13 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 22:19:18 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/10 22:19:18 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/10 22:19:18 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:19:18 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/10 22:19:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:19:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:19:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 291
2021/12/10 22:19:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 291
2021/12/10 22:19:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:19:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:19:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:19:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10650
2021/12/10 22:19:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (189.95s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/10 22:19:57 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 22:19:57 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/10 22:19:57 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:19:57 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/10 22:19:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:20:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:20:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:20:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:20:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:20:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:20:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:20:10 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:20:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 22:20:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 22:20:21 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/10 22:20:26 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/10 22:20:31 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/10 22:20:31 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:20:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:20:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:20:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:20:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:20:37 Rebalance progress: 12.5
2021/12/10 22:20:42 Rebalance progress: 12.5
2021/12/10 22:20:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:20:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:20:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:20:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:20:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:20:52 Rebalance progress: 12.5
2021/12/10 22:20:57 Rebalance progress: 12.5
2021/12/10 22:21:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:21:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:21:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:21:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 22:21:07 Rebalance progress: 12.5
2021/12/10 22:21:12 Rebalance progress: 12.5
2021/12/10 22:21:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:21:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:21:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:21:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:21:22 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 22:21:27 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 22:21:32 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 22:21:37 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/10 22:21:37 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/10 22:21:37 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:21:38 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/10 22:21:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:21:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:21:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6086
2021/12/10 22:21:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:21:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:21:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:22:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5950
2021/12/10 22:22:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5950
2021/12/10 22:22:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.31s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/10 22:22:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:22:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:22:16 Rebalance progress: 100
2021/12/10 22:22:21 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 22:22:21 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/10 22:22:21 No eventing-con process running
2021/12/10 22:22:21 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/10 22:22:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:22:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:22:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:22:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:22:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:22:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:22:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:22:35 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:22:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/10 22:22:41 Starting bucket ops to source bucket
2021/12/10 22:22:46 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/10 22:22:51 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/10 22:22:56 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/10 22:22:56 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:22:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:22:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:22:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:22:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:23:01 Rebalance progress: 12.5
2021/12/10 22:23:06 Rebalance progress: 12.5
2021/12/10 22:23:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:23:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:23:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:23:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/10 22:23:11 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/10 22:23:16 Rebalance progress: 12.5
2021/12/10 22:23:21 Rebalance progress: 12.5
2021/12/10 22:23:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:23:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:23:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:23:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:23:32 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 22:23:37 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 22:23:42 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 22:23:47 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/10 22:23:47 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/10 22:23:47 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:23:47 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/10 22:23:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:23:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:23:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4533
2021/12/10 22:23:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:24:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:24:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:24:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/10 22:24:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/10 22:24:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.11s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/10 22:24:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:24:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:24:25 Rebalance progress: 100
2021/12/10 22:24:30 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 22:24:30 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/10 22:24:30 No eventing-con process running
2021/12/10 22:24:30 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/10 22:24:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:24:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:24:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:24:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:24:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:24:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:24:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:24:44 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:24:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/10 22:24:50 Starting bucket ops to source bucket
2021/12/10 22:24:55 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/10 22:25:00 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/10 22:25:05 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/10 22:25:05 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:25:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:25:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:25:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:25:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:25:10 Rebalance progress: 12.5
2021/12/10 22:25:15 Rebalance progress: 12.5
2021/12/10 22:25:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:25:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:25:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:25:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/10 22:25:20 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/10 22:25:26 Rebalance progress: 12.5
2021/12/10 22:25:31 Rebalance progress: 12.5
2021/12/10 22:25:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:25:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:25:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:25:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:25:36 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/10 22:25:41 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 22:25:46 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 22:25:51 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 22:25:56 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/10 22:25:56 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/10 22:25:56 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:25:56 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/10 22:25:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:26:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:26:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37955
2021/12/10 22:26:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:26:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:26:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:26:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37955
2021/12/10 22:26:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (124.08s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/10 22:26:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 22:26:29 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/10 22:26:29 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:26:29 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/10 22:26:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:26:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:26:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:26:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:26:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:26:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:26:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:26:48 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:26:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:26:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:26:49 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:26:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 22:26:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 22:27:00 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/10 22:27:05 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/10 22:27:10 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/10 22:27:10 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/10 22:27:15 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/10 22:27:20 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/10 22:27:25 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/10 22:27:25 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:27:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:27:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:27:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:27:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:27:31 Rebalance progress: 12.5
2021/12/10 22:27:36 Rebalance progress: 12.5
2021/12/10 22:27:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:27:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:27:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:27:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:27:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:27:46 Rebalance progress: 12.5
2021/12/10 22:27:51 Rebalance progress: 12.5
2021/12/10 22:27:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:27:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:27:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:27:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 22:27:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:28:02 Rebalance progress: 12.5
2021/12/10 22:28:07 Rebalance progress: 12.5
2021/12/10 22:28:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:28:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:28:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:28:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:28:17 Rebalance progress: 12.5
2021/12/10 22:28:22 Rebalance progress: 12.5
2021/12/10 22:28:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:28:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:28:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:28:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 22:28:32 Rebalance progress: 12.5
2021/12/10 22:28:37 Rebalance progress: 12.5
2021/12/10 22:28:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:28:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:28:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:28:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 22:28:47 Rebalance progress: 12.5
2021/12/10 22:28:52 Rebalance progress: 12.5
2021/12/10 22:28:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:28:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:28:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:28:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:29:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 22:29:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 22:29:12 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 22:29:17 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/10 22:29:17 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/10 22:29:17 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:29:18 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/10 22:29:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:29:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:29:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2136
2021/12/10 22:29:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2136
2021/12/10 22:29:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:29:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:29:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:29:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12221
2021/12/10 22:29:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:29:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:29:53 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 22:29:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 22:30:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 22:30:08 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/10 22:30:08 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/10 22:30:08 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:30:08 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/10 22:30:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:30:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:30:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:30:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:30:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:30:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (242.05s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/10 22:30:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 22:30:31 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/10 22:30:31 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:30:31 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/10 22:30:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:30:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:30:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:30:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:30:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:30:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:30:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:30:50 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:30:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:30:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:30:51 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:30:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 22:30:57 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 22:31:02 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/10 22:31:07 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/10 22:31:12 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/10 22:31:12 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/10 22:31:17 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/10 22:31:22 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/10 22:31:27 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/10 22:31:27 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:31:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:31:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:31:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:31:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:31:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/10 22:31:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:31:33 Rebalance progress: 12.5
2021/12/10 22:31:38 Rebalance progress: 12.5
2021/12/10 22:31:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:31:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:31:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:31:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 22:31:48 Rebalance progress: 12.5
2021/12/10 22:31:53 Rebalance progress: 12.5
2021/12/10 22:31:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:31:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:31:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:31:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:32:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 22:32:09 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 22:32:14 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 22:32:19 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/10 22:32:19 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/10 22:32:19 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:32:19 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/10 22:32:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:32:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:32:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4353
2021/12/10 22:32:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4353
2021/12/10 22:32:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:32:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:32:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:32:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6117
2021/12/10 22:32:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:32:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:32:54 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 22:32:59 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 22:33:04 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 22:33:09 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/10 22:33:09 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/10 22:33:09 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:33:09 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/10 22:33:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:33:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:33:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:33:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:33:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:33:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.03s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/10 22:33:32 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 22:33:32 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/10 22:33:32 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:33:32 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/10 22:33:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:33:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:33:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:33:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:33:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:33:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:33:51 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 22:33:51 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/10 22:33:51 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:33:51 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/10 22:33:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:33:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:33:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:33:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:34:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:34:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:34:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:34:04 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:34:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:34:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:34:05 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:34:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/10 22:34:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 22:34:17 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/10 22:34:22 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/10 22:34:27 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/10 22:34:27 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/10 22:34:32 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/10 22:34:37 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/10 22:34:42 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/10 22:34:42 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:34:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:34:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:34:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:34:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:34:47 Rebalance progress: 12.5
2021/12/10 22:34:52 Rebalance progress: 12.5
2021/12/10 22:34:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:34:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:34:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:34:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:34:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:35:03 Rebalance progress: 12.5
2021/12/10 22:35:08 Rebalance progress: 12.5
2021/12/10 22:35:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:35:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:35:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:35:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 22:35:18 Rebalance progress: 12.5
2021/12/10 22:35:23 Rebalance progress: 12.5
2021/12/10 22:35:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:35:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:35:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:35:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:35:33 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 22:35:38 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 22:35:43 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 22:35:48 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/10 22:35:48 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/10 22:35:48 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:35:48 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/10 22:35:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:35:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:36:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4823
2021/12/10 22:36:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4823
2021/12/10 22:36:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:36:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:36:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:36:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7653
2021/12/10 22:36:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:36:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:36:23 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 22:36:28 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 22:36:33 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 22:36:38 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/10 22:36:38 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/10 22:36:38 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:36:38 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/10 22:36:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:36:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:36:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:36:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:36:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:36:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (209.68s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/10 22:37:02 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:37:02 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 22:37:02 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:37:02 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/10 22:37:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:37:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:37:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:37:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:37:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:37:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:37:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:37:21 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:37:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/10 22:37:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 22:37:32 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 22:37:37 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 22:37:42 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 22:37:42 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:37:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:37:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:37:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:37:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:38:02 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:38:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:38:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:38:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:38:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:38:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:38:38 Rebalance progress: 12.5
2021/12/10 22:38:43 Rebalance progress: 12.5
2021/12/10 22:38:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:38:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:38:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:38:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:38:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:38:53 Rebalance progress: 12.5
2021/12/10 22:38:58 Rebalance progress: 12.5
2021/12/10 22:39:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:39:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:39:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:39:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 22:39:08 Rebalance progress: 12.5
2021/12/10 22:39:13 Rebalance progress: 12.5
2021/12/10 22:39:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:39:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:39:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:39:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:39:24 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:39:29 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:39:34 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:39:39 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:39:39 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 22:39:39 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:39:39 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/10 22:39:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:39:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:39:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4845
2021/12/10 22:39:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4845
2021/12/10 22:39:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:39:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:40:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:40:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11167
2021/12/10 22:40:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.14s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/10 22:40:17 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:40:17 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 22:40:17 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:40:17 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/10 22:40:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:40:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:40:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:40:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:40:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:40:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:40:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:40:36 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:40:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/10 22:40:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 22:40:47 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 22:40:52 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 22:40:57 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/10 22:40:57 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:40:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:40:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:40:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:40:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:41:17 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:41:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:41:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:41:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:41:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:41:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:41:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:41:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:41:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:41:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:41:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:41:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:41:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:41:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:42:19 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:42:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:42:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:42:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:42:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:42:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:42:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:42:50 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:42:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:42:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:43:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:43:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:43:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:43:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:43:21 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:43:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:43:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:43:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:43:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:43:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:43:36 Rebalance progress: 12.5
2021/12/10 22:43:41 Rebalance progress: 12.5
2021/12/10 22:43:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:43:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:43:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:43:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:43:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:43:52 Rebalance progress: 12.5
2021/12/10 22:43:57 Rebalance progress: 12.5
2021/12/10 22:44:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:44:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:44:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:44:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 22:44:07 Rebalance progress: 12.5
2021/12/10 22:44:12 Rebalance progress: 12.5
2021/12/10 22:44:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:44:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:44:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:44:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:44:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:44:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:44:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:44:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/10 22:44:37 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/10 22:44:37 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:44:37 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/10 22:44:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:44:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:44:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21500
2021/12/10 22:44:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:44:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:44:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:45:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 13381
2021/12/10 22:45:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 13381
2021/12/10 22:45:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (298.47s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/10 22:45:16 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 22:45:16 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/10 22:45:16 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:45:16 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/10 22:45:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:45:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:45:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:45:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:45:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:45:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:45:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:45:34 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:45:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 22:45:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 22:45:45 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/10 22:45:50 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/10 22:45:55 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/10 22:45:55 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:45:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:45:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:45:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:45:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:46:01 Rebalance progress: 12.5
2021/12/10 22:46:06 Rebalance progress: 12.5
2021/12/10 22:46:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:46:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:46:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:46:11 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:46:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:46:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:46:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:46:26 Rebalance progress: 12.5
2021/12/10 22:46:31 Rebalance progress: 12.5
2021/12/10 22:46:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:46:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:46:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:46:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 22:46:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:46:41 Rebalance progress: 12.5
2021/12/10 22:46:46 Rebalance progress: 12.5
2021/12/10 22:46:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:46:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:46:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:46:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 22:46:56 Rebalance progress: 12.5
2021/12/10 22:47:01 Rebalance progress: 12.5
2021/12/10 22:47:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:47:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:47:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:47:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:47:12 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 22:47:17 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 22:47:22 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 22:47:27 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/10 22:47:27 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/10 22:47:27 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:47:27 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/10 22:47:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:47:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:47:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8606
2021/12/10 22:47:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:47:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:47:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:47:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1651
2021/12/10 22:47:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1651
2021/12/10 22:47:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.53s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/10 22:48:05 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 22:48:05 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/10 22:48:05 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:48:05 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/10 22:48:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:48:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:48:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:48:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:48:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:48:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:48:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:48:24 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:48:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/10 22:48:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 22:48:35 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/10 22:48:40 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/10 22:48:45 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/10 22:48:45 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:48:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:48:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:48:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/10 22:48:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:48:51 Rebalance progress: 12.5
2021/12/10 22:48:56 Rebalance progress: 12.5
2021/12/10 22:49:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:49:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:49:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:49:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/10 22:49:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:49:06 Rebalance progress: 12.5
2021/12/10 22:49:11 Rebalance progress: 12.5
2021/12/10 22:49:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:49:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:49:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:49:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 22:49:21 Rebalance progress: 12.5
2021/12/10 22:49:26 Rebalance progress: 12.5
2021/12/10 22:49:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:49:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:49:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:49:32 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:49:37 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 22:49:42 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 22:49:47 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 22:49:52 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/10 22:49:52 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/10 22:49:52 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:49:52 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/10 22:49:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:49:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:50:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6170
2021/12/10 22:50:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6170
2021/12/10 22:50:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:50:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:50:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:50:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1851
2021/12/10 22:50:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1851
2021/12/10 22:50:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (150.13s)
=== RUN   TestEventingRebInWithOps
2021/12/10 22:50:35 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 22:50:35 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/10 22:50:35 eventingSetup.go:46: No eventing-con process running
2021/12/10 22:50:35 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/10 22:50:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:50:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:50:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:50:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:50:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:50:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:50:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:50:54 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:50:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 22:51:00 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 22:51:05 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 22:51:10 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 22:51:15 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/10 22:51:15 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/10 22:51:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:51:15 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 22:51:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 22:51:21 Rebalance progress: 12.5
2021/12/10 22:51:26 Rebalance progress: 12.5
2021/12/10 22:51:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:51:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:51:36 Rebalance progress: 12.5
2021/12/10 22:51:41 Rebalance progress: 12.5
2021/12/10 22:51:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:51:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:51:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:51:46 Waiting for src bucket item count to get to: 200000 curr count: 37173
2021/12/10 22:51:51 Waiting for src bucket item count to get to: 200000 curr count: 53169
2021/12/10 22:51:56 Waiting for src bucket item count to get to: 200000 curr count: 53169
2021/12/10 22:52:01 Waiting for src bucket item count to get to: 200000 curr count: 69202
2021/12/10 22:52:06 Waiting for src bucket item count to get to: 200000 curr count: 69202
2021/12/10 22:52:11 Waiting for src bucket item count to get to: 200000 curr count: 84069
2021/12/10 22:52:16 Waiting for src bucket item count to get to: 200000 curr count: 84069
2021/12/10 22:52:21 Waiting for src bucket item count to get to: 200000 curr count: 100143
2021/12/10 22:52:26 Waiting for src bucket item count to get to: 200000 curr count: 100143
2021/12/10 22:52:31 Waiting for src bucket item count to get to: 200000 curr count: 116113
2021/12/10 22:52:36 Waiting for src bucket item count to get to: 200000 curr count: 116113
2021/12/10 22:52:41 Waiting for src bucket item count to get to: 200000 curr count: 132103
2021/12/10 22:52:46 Waiting for src bucket item count to get to: 200000 curr count: 132103
2021/12/10 22:52:51 Waiting for src bucket item count to get to: 200000 curr count: 132103
2021/12/10 22:52:56 Waiting for src bucket item count to get to: 200000 curr count: 148260
2021/12/10 22:53:01 Waiting for src bucket item count to get to: 200000 curr count: 148260
2021/12/10 22:53:06 Waiting for src bucket item count to get to: 200000 curr count: 164547
2021/12/10 22:53:11 Waiting for src bucket item count to get to: 200000 curr count: 164547
2021/12/10 22:53:16 Waiting for src bucket item count to get to: 200000 curr count: 179458
2021/12/10 22:53:21 Waiting for src bucket item count to get to: 200000 curr count: 179458
2021/12/10 22:53:26 Waiting for src bucket item count to get to: 200000 curr count: 195385
2021/12/10 22:53:31 Waiting for src bucket item count to get to: 200000 curr count: 195385
2021/12/10 22:53:36 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 22:53:36 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 22:53:36 Starting bucket ops to source bucket
2021/12/10 22:57:21 Waiting for dst bucket item count to get to: 0 curr count: 12061
2021/12/10 22:57:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:57:21 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 22:57:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 22:57:26 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 22:57:31 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 22:57:36 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 22:57:41 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/10 22:57:41 App: TestEventingRebInWithOps got un-deployed
2021/12/10 22:57:41 No eventing-con process running
2021/12/10 22:57:41 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/10 22:57:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:57:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:57:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:57:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:57:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:57:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (449.19s)
=== RUN   TestEventingRebOutWithOps
2021/12/10 22:57:59 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 22:58:00 Removing node(s): [] from the cluster
2021/12/10 22:58:05 Rebalance progress: 100
2021/12/10 22:58:10 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 22:58:10 App: TestEventingRebOutWithOps got un-deployed
2021/12/10 22:58:10 No eventing-con process running
2021/12/10 22:58:10 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/10 22:58:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:58:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:58:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 22:58:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:58:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:58:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:58:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 22:58:28 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 22:58:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 22:58:34 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 22:58:39 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 22:58:44 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 22:58:49 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/10 22:58:49 App: TestEventingRebOutWithOps got deployed
2021/12/10 22:58:49 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 22:58:49 Starting bucket ops to source bucket
2021/12/10 22:58:54 Rebalance progress: 12.5
2021/12/10 22:58:59 Rebalance progress: 12.5
2021/12/10 22:59:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:59:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 22:59:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 22:59:04 Waiting for src bucket item count to get to: 200000 curr count: 15351
2021/12/10 22:59:09 Waiting for src bucket item count to get to: 200000 curr count: 15351
2021/12/10 22:59:14 Waiting for src bucket item count to get to: 200000 curr count: 31543
2021/12/10 22:59:19 Waiting for src bucket item count to get to: 200000 curr count: 31543
2021/12/10 22:59:24 Waiting for src bucket item count to get to: 200000 curr count: 48155
2021/12/10 22:59:29 Waiting for src bucket item count to get to: 200000 curr count: 48155
2021/12/10 22:59:34 Waiting for src bucket item count to get to: 200000 curr count: 48155
2021/12/10 22:59:39 Waiting for src bucket item count to get to: 200000 curr count: 64425
2021/12/10 22:59:44 Waiting for src bucket item count to get to: 200000 curr count: 64425
2021/12/10 22:59:49 Waiting for src bucket item count to get to: 200000 curr count: 79707
2021/12/10 22:59:54 Waiting for src bucket item count to get to: 200000 curr count: 95985
2021/12/10 22:59:59 Waiting for src bucket item count to get to: 200000 curr count: 95985
2021/12/10 23:00:05 Waiting for src bucket item count to get to: 200000 curr count: 112337
2021/12/10 23:00:10 Waiting for src bucket item count to get to: 200000 curr count: 112337
2021/12/10 23:00:15 Waiting for src bucket item count to get to: 200000 curr count: 128316
2021/12/10 23:00:20 Waiting for src bucket item count to get to: 200000 curr count: 128316
2021/12/10 23:00:25 Waiting for src bucket item count to get to: 200000 curr count: 144571
2021/12/10 23:00:30 Waiting for src bucket item count to get to: 200000 curr count: 144571
2021/12/10 23:00:35 Waiting for src bucket item count to get to: 200000 curr count: 160703
2021/12/10 23:00:40 Waiting for src bucket item count to get to: 200000 curr count: 160703
2021/12/10 23:00:45 Waiting for src bucket item count to get to: 200000 curr count: 175734
2021/12/10 23:00:50 Waiting for src bucket item count to get to: 200000 curr count: 175734
2021/12/10 23:00:55 Waiting for src bucket item count to get to: 200000 curr count: 191881
2021/12/10 23:01:00 Waiting for src bucket item count to get to: 200000 curr count: 191881
2021/12/10 23:01:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 23:01:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 23:01:05 Starting bucket ops to source bucket
2021/12/10 23:04:49 Waiting for dst bucket item count to get to: 0 curr count: 27131
2021/12/10 23:04:54 Waiting for dst bucket item count to get to: 0 curr count: 8936
2021/12/10 23:04:59 Waiting for dst bucket item count to get to: 0 curr count: 8936
2021/12/10 23:04:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:04:59 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 23:04:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 23:05:04 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 23:05:09 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 23:05:14 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 23:05:19 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/10 23:05:19 App: TestEventingRebOutWithOps got un-deployed
2021/12/10 23:05:19 No eventing-con process running
2021/12/10 23:05:19 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/10 23:05:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:05:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:05:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:05:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:05:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:05:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (458.50s)
=== RUN   TestEventingSwapRebWithOps
2021/12/10 23:05:43 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 23:05:43 App: TestEventingSwapRebWithOps got un-deployed
2021/12/10 23:05:43 No eventing-con process running
2021/12/10 23:05:43 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/10 23:05:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:05:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:05:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:05:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:05:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:05:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:06:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:06:01 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 23:06:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 23:06:08 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 23:06:13 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 23:06:18 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 23:06:23 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/10 23:06:23 App: TestEventingSwapRebWithOps got deployed
2021/12/10 23:06:23 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 23:06:23 Starting bucket ops to source bucket
2021/12/10 23:06:23 Removing node(s): [] from the cluster
2021/12/10 23:06:28 Rebalance progress: 12.5
2021/12/10 23:06:33 Rebalance progress: 12.5
2021/12/10 23:06:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:06:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:06:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:06:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 23:06:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 23:06:44 Rebalance progress: 12.5
2021/12/10 23:06:49 Rebalance progress: 12.5
2021/12/10 23:06:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:06:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:06:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:06:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 23:06:59 Rebalance progress: 12.5
2021/12/10 23:07:04 Rebalance progress: 12.5
2021/12/10 23:07:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:07:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:07:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:07:09 Waiting for src bucket item count to get to: 200000 curr count: 56281
2021/12/10 23:07:14 Waiting for src bucket item count to get to: 200000 curr count: 71305
2021/12/10 23:07:19 Waiting for src bucket item count to get to: 200000 curr count: 71305
2021/12/10 23:07:24 Waiting for src bucket item count to get to: 200000 curr count: 87083
2021/12/10 23:07:29 Waiting for src bucket item count to get to: 200000 curr count: 87083
2021/12/10 23:07:34 Waiting for src bucket item count to get to: 200000 curr count: 103279
2021/12/10 23:07:39 Waiting for src bucket item count to get to: 200000 curr count: 103279
2021/12/10 23:07:44 Waiting for src bucket item count to get to: 200000 curr count: 119716
2021/12/10 23:07:49 Waiting for src bucket item count to get to: 200000 curr count: 119716
2021/12/10 23:07:54 Waiting for src bucket item count to get to: 200000 curr count: 136152
2021/12/10 23:07:59 Waiting for src bucket item count to get to: 200000 curr count: 136152
2021/12/10 23:08:04 Waiting for src bucket item count to get to: 200000 curr count: 152591
2021/12/10 23:08:09 Waiting for src bucket item count to get to: 200000 curr count: 152591
2021/12/10 23:08:14 Waiting for src bucket item count to get to: 200000 curr count: 168571
2021/12/10 23:08:19 Waiting for src bucket item count to get to: 200000 curr count: 168571
2021/12/10 23:08:24 Waiting for src bucket item count to get to: 200000 curr count: 184183
2021/12/10 23:08:29 Waiting for src bucket item count to get to: 200000 curr count: 184183
2021/12/10 23:08:34 Waiting for src bucket item count to get to: 200000 curr count: 184183
2021/12/10 23:08:39 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 23:08:39 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 23:08:39 Starting bucket ops to source bucket
2021/12/10 23:12:24 Waiting for dst bucket item count to get to: 0 curr count: 17656
2021/12/10 23:12:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:12:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 23:12:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 23:12:29 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 23:12:34 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 23:12:39 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 23:12:44 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/10 23:12:44 App: TestEventingSwapRebWithOps got un-deployed
2021/12/10 23:12:44 No eventing-con process running
2021/12/10 23:12:44 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/10 23:12:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:12:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:12:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:12:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:12:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:12:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (444.70s)
=== RUN   TestKVRebInWithOps
2021/12/10 23:13:08 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 23:13:08 App: TestKVRebInWithOps got un-deployed
2021/12/10 23:13:08 No eventing-con process running
2021/12/10 23:13:08 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/10 23:13:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:13:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:13:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:13:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:13:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:13:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:13:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:13:26 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 23:13:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 23:13:32 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 23:13:37 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 23:13:42 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 23:13:47 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/10 23:13:47 App: TestKVRebInWithOps got deployed
2021/12/10 23:13:47 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 23:13:47 Starting bucket ops to source bucket
2021/12/10 23:13:48 Removing node(s): [] from the cluster
2021/12/10 23:13:53 Rebalance progress: 12.5
2021/12/10 23:13:58 Rebalance progress: 12.5
2021/12/10 23:14:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:14:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:14:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:14:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 23:14:08 Rebalance progress: 12.5
2021/12/10 23:14:13 Rebalance progress: 12.5
2021/12/10 23:14:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:14:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:14:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:14:18 Waiting for src bucket item count to get to: 200000 curr count: 33950
2021/12/10 23:14:23 Waiting for src bucket item count to get to: 200000 curr count: 50348
2021/12/10 23:14:28 Waiting for src bucket item count to get to: 200000 curr count: 50348
2021/12/10 23:14:33 Waiting for src bucket item count to get to: 200000 curr count: 66669
2021/12/10 23:14:38 Waiting for src bucket item count to get to: 200000 curr count: 66669
2021/12/10 23:14:43 Waiting for src bucket item count to get to: 200000 curr count: 81667
2021/12/10 23:14:48 Waiting for src bucket item count to get to: 200000 curr count: 81667
2021/12/10 23:14:53 Waiting for src bucket item count to get to: 200000 curr count: 98214
2021/12/10 23:14:58 Waiting for src bucket item count to get to: 200000 curr count: 98214
2021/12/10 23:15:03 Waiting for src bucket item count to get to: 200000 curr count: 98214
2021/12/10 23:15:08 Waiting for src bucket item count to get to: 200000 curr count: 114611
2021/12/10 23:15:13 Waiting for src bucket item count to get to: 200000 curr count: 114611
2021/12/10 23:15:18 Waiting for src bucket item count to get to: 200000 curr count: 131178
2021/12/10 23:15:23 Waiting for src bucket item count to get to: 200000 curr count: 131178
2021/12/10 23:15:28 Waiting for src bucket item count to get to: 200000 curr count: 147767
2021/12/10 23:15:33 Waiting for src bucket item count to get to: 200000 curr count: 147767
2021/12/10 23:15:38 Waiting for src bucket item count to get to: 200000 curr count: 164069
2021/12/10 23:15:43 Waiting for src bucket item count to get to: 200000 curr count: 164069
2021/12/10 23:15:48 Waiting for src bucket item count to get to: 200000 curr count: 179280
2021/12/10 23:15:53 Waiting for src bucket item count to get to: 200000 curr count: 179280
2021/12/10 23:15:58 Waiting for src bucket item count to get to: 200000 curr count: 195473
2021/12/10 23:16:03 Waiting for src bucket item count to get to: 200000 curr count: 195473
2021/12/10 23:16:08 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 23:16:08 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 23:16:08 Starting bucket ops to source bucket
2021/12/10 23:19:53 Waiting for dst bucket item count to get to: 0 curr count: 15450
2021/12/10 23:19:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:19:53 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 23:19:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 23:19:58 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 23:20:03 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 23:20:08 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 23:20:13 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/10 23:20:13 App: TestKVRebInWithOps got un-deployed
2021/12/10 23:20:13 No eventing-con process running
2021/12/10 23:20:13 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/10 23:20:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:20:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:20:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:20:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:20:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:20:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (448.59s)
=== RUN   TestKVRebOutWithOps
2021/12/10 23:20:31 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 23:20:31 Removing node(s): [] from the cluster
2021/12/10 23:20:36 Rebalance progress: 100
2021/12/10 23:20:41 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 23:20:41 App: TestKVRebOutWithOps got un-deployed
2021/12/10 23:20:41 No eventing-con process running
2021/12/10 23:20:41 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/10 23:20:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:20:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:20:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:20:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:20:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:21:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:21:00 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 23:21:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 23:21:06 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 23:21:11 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 23:21:16 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 23:21:21 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/10 23:21:21 App: TestKVRebOutWithOps got deployed
2021/12/10 23:21:21 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 23:21:21 Starting bucket ops to source bucket
2021/12/10 23:21:26 Rebalance progress: 12.5
2021/12/10 23:21:31 Rebalance progress: 12.5
2021/12/10 23:21:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:21:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:21:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:21:36 Waiting for src bucket item count to get to: 200000 curr count: 12214
2021/12/10 23:21:41 Waiting for src bucket item count to get to: 200000 curr count: 28409
2021/12/10 23:21:46 Waiting for src bucket item count to get to: 200000 curr count: 28409
2021/12/10 23:21:51 Waiting for src bucket item count to get to: 200000 curr count: 44717
2021/12/10 23:21:56 Waiting for src bucket item count to get to: 200000 curr count: 44717
2021/12/10 23:22:01 Waiting for src bucket item count to get to: 200000 curr count: 61327
2021/12/10 23:22:06 Waiting for src bucket item count to get to: 200000 curr count: 61327
2021/12/10 23:22:11 Waiting for src bucket item count to get to: 200000 curr count: 61327
2021/12/10 23:22:16 Waiting for src bucket item count to get to: 200000 curr count: 76769
2021/12/10 23:22:21 Waiting for src bucket item count to get to: 200000 curr count: 76769
2021/12/10 23:22:26 Waiting for src bucket item count to get to: 200000 curr count: 92972
2021/12/10 23:22:31 Waiting for src bucket item count to get to: 200000 curr count: 92972
2021/12/10 23:22:36 Waiting for src bucket item count to get to: 200000 curr count: 109528
2021/12/10 23:22:41 Waiting for src bucket item count to get to: 200000 curr count: 109528
2021/12/10 23:22:46 Waiting for src bucket item count to get to: 200000 curr count: 125981
2021/12/10 23:22:51 Waiting for src bucket item count to get to: 200000 curr count: 125981
2021/12/10 23:22:56 Waiting for src bucket item count to get to: 200000 curr count: 142543
2021/12/10 23:23:01 Waiting for src bucket item count to get to: 200000 curr count: 142543
2021/12/10 23:23:06 Waiting for src bucket item count to get to: 200000 curr count: 159199
2021/12/10 23:23:11 Waiting for src bucket item count to get to: 200000 curr count: 159199
2021/12/10 23:23:16 Waiting for src bucket item count to get to: 200000 curr count: 174510
2021/12/10 23:23:21 Waiting for src bucket item count to get to: 200000 curr count: 174510
2021/12/10 23:23:26 Waiting for src bucket item count to get to: 200000 curr count: 190582
2021/12/10 23:23:31 Waiting for src bucket item count to get to: 200000 curr count: 190582
2021/12/10 23:23:36 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 23:23:36 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 23:23:36 Starting bucket ops to source bucket
2021/12/10 23:27:20 Waiting for dst bucket item count to get to: 0 curr count: 11372
2021/12/10 23:27:25 Waiting for dst bucket item count to get to: 0 curr count: 11372
2021/12/10 23:27:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:27:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 23:27:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 23:27:31 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 23:27:36 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 23:27:41 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 23:27:46 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 23:27:46 App: TestKVRebOutWithOps got un-deployed
2021/12/10 23:27:46 No eventing-con process running
2021/12/10 23:27:46 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/10 23:27:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:27:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:27:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:27:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:27:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:27:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (452.98s)
=== RUN   TestKVSwapRebWithOps
2021/12/10 23:28:09 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 23:28:09 App: TestKVSwapRebWithOps got un-deployed
2021/12/10 23:28:09 No eventing-con process running
2021/12/10 23:28:09 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/10 23:28:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:28:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:28:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:28:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:28:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:28:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:28:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:28:28 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 23:28:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 23:28:34 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 23:28:39 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 23:28:44 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 23:28:49 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 23:28:49 App: TestKVSwapRebWithOps got deployed
2021/12/10 23:28:49 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 23:28:49 Starting bucket ops to source bucket
2021/12/10 23:28:49 Removing node(s): [] from the cluster
2021/12/10 23:28:54 Rebalance progress: 12.5
2021/12/10 23:28:59 Rebalance progress: 12.5
2021/12/10 23:29:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:29:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:29:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:29:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 23:29:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 23:29:09 Rebalance progress: 12.5
2021/12/10 23:29:14 Rebalance progress: 12.5
2021/12/10 23:29:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:29:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:29:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:29:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 23:29:25 Rebalance progress: 12.5
2021/12/10 23:29:30 Rebalance progress: 12.5
2021/12/10 23:29:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:29:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:29:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:29:35 Waiting for src bucket item count to get to: 200000 curr count: 62565
2021/12/10 23:29:40 Waiting for src bucket item count to get to: 200000 curr count: 62565
2021/12/10 23:29:45 Waiting for src bucket item count to get to: 200000 curr count: 77577
2021/12/10 23:29:50 Waiting for src bucket item count to get to: 200000 curr count: 77577
2021/12/10 23:29:55 Waiting for src bucket item count to get to: 200000 curr count: 93598
2021/12/10 23:30:00 Waiting for src bucket item count to get to: 200000 curr count: 93598
2021/12/10 23:30:05 Waiting for src bucket item count to get to: 200000 curr count: 109697
2021/12/10 23:30:10 Waiting for src bucket item count to get to: 200000 curr count: 109697
2021/12/10 23:30:15 Waiting for src bucket item count to get to: 200000 curr count: 126043
2021/12/10 23:30:20 Waiting for src bucket item count to get to: 200000 curr count: 126043
2021/12/10 23:30:25 Waiting for src bucket item count to get to: 200000 curr count: 142667
2021/12/10 23:30:30 Waiting for src bucket item count to get to: 200000 curr count: 142667
2021/12/10 23:30:35 Waiting for src bucket item count to get to: 200000 curr count: 158904
2021/12/10 23:30:40 Waiting for src bucket item count to get to: 200000 curr count: 158904
2021/12/10 23:30:45 Waiting for src bucket item count to get to: 200000 curr count: 174291
2021/12/10 23:30:50 Waiting for src bucket item count to get to: 200000 curr count: 174291
2021/12/10 23:30:55 Waiting for src bucket item count to get to: 200000 curr count: 190174
2021/12/10 23:31:00 Waiting for src bucket item count to get to: 200000 curr count: 190174
2021/12/10 23:31:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 23:31:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 23:31:05 Starting bucket ops to source bucket
2021/12/10 23:34:50 Waiting for dst bucket item count to get to: 0 curr count: 9633
2021/12/10 23:34:55 Waiting for dst bucket item count to get to: 0 curr count: 9633
2021/12/10 23:34:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:34:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 23:34:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 23:35:00 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 23:35:05 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 23:35:10 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 23:35:15 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 23:35:15 App: TestKVSwapRebWithOps got un-deployed
2021/12/10 23:35:15 No eventing-con process running
2021/12/10 23:35:15 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/10 23:35:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:35:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:35:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:35:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:35:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:35:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (449.07s)
=== RUN   TestEventingRebInWithTimers
2021/12/10 23:35:38 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 23:35:38 App: TestEventingRebInWithTimers got un-deployed
2021/12/10 23:35:38 No eventing-con process running
2021/12/10 23:35:38 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/10 23:35:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:35:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:35:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:35:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:35:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:35:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:35:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:35:57 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 23:35:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 23:36:03 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 23:36:08 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 23:36:13 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 23:36:18 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 23:36:18 App: TestEventingRebInWithTimers got deployed
2021/12/10 23:36:18 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 23:36:18 Starting bucket ops to source bucket
2021/12/10 23:36:19 Removing node(s): [] from the cluster
2021/12/10 23:36:24 Rebalance progress: 12.5
2021/12/10 23:36:29 Rebalance progress: 12.5
2021/12/10 23:36:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:36:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:36:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:36:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 23:36:39 Rebalance progress: 12.5
2021/12/10 23:36:44 Rebalance progress: 12.5
2021/12/10 23:36:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:36:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:36:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:36:49 Waiting for src bucket item count to get to: 50000 curr count: 28786
2021/12/10 23:36:54 Waiting for src bucket item count to get to: 50000 curr count: 38787
2021/12/10 23:36:59 Waiting for src bucket item count to get to: 50000 curr count: 38787
2021/12/10 23:37:09 Waiting for dst bucket item count to get to: 50000 curr count: 28576
2021/12/10 23:37:14 Waiting for dst bucket item count to get to: 50000 curr count: 28576
2021/12/10 23:37:19 Waiting for dst bucket item count to get to: 50000 curr count: 43040
2021/12/10 23:37:24 Waiting for dst bucket item count to get to: 50000 curr count: 43040
2021/12/10 23:37:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 23:37:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 23:37:24 Starting bucket ops to source bucket
2021/12/10 23:38:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:38:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:38:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:38:48 Waiting for dst bucket item count to get to: 0 curr count: 41415
2021/12/10 23:38:53 Waiting for dst bucket item count to get to: 0 curr count: 41415
2021/12/10 23:38:58 Waiting for dst bucket item count to get to: 0 curr count: 32074
2021/12/10 23:39:03 Waiting for dst bucket item count to get to: 0 curr count: 32074
2021/12/10 23:39:08 Waiting for dst bucket item count to get to: 0 curr count: 17070
2021/12/10 23:39:13 Waiting for dst bucket item count to get to: 0 curr count: 17070
2021/12/10 23:39:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:39:13 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 23:39:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 23:39:18 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 23:39:23 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 23:39:28 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 23:39:33 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 23:39:33 App: TestEventingRebInWithTimers got un-deployed
2021/12/10 23:39:33 No eventing-con process running
2021/12/10 23:39:33 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/10 23:39:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:39:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:39:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:39:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:39:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:39:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (258.62s)
=== RUN   TestEventingRebOutWithTimers
2021/12/10 23:39:52 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 23:39:52 Removing node(s): [] from the cluster
2021/12/10 23:39:57 Rebalance progress: 100
2021/12/10 23:40:02 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 23:40:02 App: TestEventingRebOutWithTimers got un-deployed
2021/12/10 23:40:02 No eventing-con process running
2021/12/10 23:40:02 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/10 23:40:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:40:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:40:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:40:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:40:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:40:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:40:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:40:21 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 23:40:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 23:40:27 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 23:40:32 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 23:40:37 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 23:40:42 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 23:40:42 App: TestEventingRebOutWithTimers got deployed
2021/12/10 23:40:42 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 23:40:42 Starting bucket ops to source bucket
2021/12/10 23:40:47 Rebalance progress: 12.5
2021/12/10 23:40:52 Rebalance progress: 12.5
2021/12/10 23:40:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:40:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:40:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:40:57 Waiting for src bucket item count to get to: 50000 curr count: 10282
2021/12/10 23:41:02 Waiting for src bucket item count to get to: 50000 curr count: 24269
2021/12/10 23:41:07 Waiting for src bucket item count to get to: 50000 curr count: 24269
2021/12/10 23:41:12 Waiting for src bucket item count to get to: 50000 curr count: 35765
2021/12/10 23:41:17 Waiting for src bucket item count to get to: 50000 curr count: 35765
2021/12/10 23:41:22 Waiting for src bucket item count to get to: 50000 curr count: 35765
2021/12/10 23:41:27 Waiting for src bucket item count to get to: 50000 curr count: 46476
2021/12/10 23:41:32 Waiting for src bucket item count to get to: 50000 curr count: 46476
2021/12/10 23:41:42 Waiting for dst bucket item count to get to: 50000 curr count: 37172
2021/12/10 23:41:47 Waiting for dst bucket item count to get to: 50000 curr count: 37172
2021/12/10 23:41:47 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 23:41:47 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 23:41:47 Starting bucket ops to source bucket
2021/12/10 23:42:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:43:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:43:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:43:11 Waiting for dst bucket item count to get to: 0 curr count: 49938
2021/12/10 23:43:16 Waiting for dst bucket item count to get to: 0 curr count: 49938
2021/12/10 23:43:21 Waiting for dst bucket item count to get to: 0 curr count: 34157
2021/12/10 23:43:26 Waiting for dst bucket item count to get to: 0 curr count: 34157
2021/12/10 23:43:31 Waiting for dst bucket item count to get to: 0 curr count: 17010
2021/12/10 23:43:36 Waiting for dst bucket item count to get to: 0 curr count: 17010
2021/12/10 23:43:41 Waiting for dst bucket item count to get to: 0 curr count: 3839
2021/12/10 23:43:46 Waiting for dst bucket item count to get to: 0 curr count: 3839
2021/12/10 23:43:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:43:46 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 23:43:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 23:43:51 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 23:43:56 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 23:44:01 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 23:44:06 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 23:44:06 App: TestEventingRebOutWithTimers got un-deployed
2021/12/10 23:44:06 No eventing-con process running
2021/12/10 23:44:06 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/10 23:44:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:44:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:44:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:44:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:44:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:44:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (273.07s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/10 23:44:30 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 23:44:30 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/10 23:44:30 No eventing-con process running
2021/12/10 23:44:30 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/10 23:44:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:44:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:44:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:44:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:44:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:44:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:44:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:44:48 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 23:44:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 23:44:55 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 23:45:00 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 23:45:05 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 23:45:10 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 23:45:10 App: TestEventingSwapRebWithTimers got deployed
2021/12/10 23:45:10 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 23:45:10 Starting bucket ops to source bucket
2021/12/10 23:45:10 Removing node(s): [] from the cluster
2021/12/10 23:45:15 Rebalance progress: 12.5
2021/12/10 23:45:20 Rebalance progress: 12.5
2021/12/10 23:45:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:45:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:45:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:45:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 23:45:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 23:45:31 Rebalance progress: 12.5
2021/12/10 23:45:36 Rebalance progress: 12.5
2021/12/10 23:45:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:45:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:45:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:45:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 23:45:46 Rebalance progress: 12.5
2021/12/10 23:45:51 Rebalance progress: 12.5
2021/12/10 23:45:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:45:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:45:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:45:56 Waiting for src bucket item count to get to: 50000 curr count: 47864
2021/12/10 23:46:06 Waiting for dst bucket item count to get to: 50000 curr count: 40969
2021/12/10 23:46:11 Waiting for dst bucket item count to get to: 50000 curr count: 40969
2021/12/10 23:46:16 Waiting for dst bucket item count to get to: 50000 curr count: 49996
2021/12/10 23:46:21 Waiting for dst bucket item count to get to: 50000 curr count: 49996
2021/12/10 23:46:21 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 23:46:21 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 23:46:21 Starting bucket ops to source bucket
2021/12/10 23:47:30 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:47:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:47:40 Waiting for dst bucket item count to get to: 0 curr count: 48523
2021/12/10 23:47:45 Waiting for dst bucket item count to get to: 0 curr count: 48523
2021/12/10 23:47:50 Waiting for dst bucket item count to get to: 0 curr count: 38753
2021/12/10 23:47:55 Waiting for dst bucket item count to get to: 0 curr count: 38753
2021/12/10 23:48:00 Waiting for dst bucket item count to get to: 0 curr count: 28722
2021/12/10 23:48:05 Waiting for dst bucket item count to get to: 0 curr count: 28722
2021/12/10 23:48:10 Waiting for dst bucket item count to get to: 0 curr count: 7809
2021/12/10 23:48:15 Waiting for dst bucket item count to get to: 0 curr count: 7809
2021/12/10 23:48:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:48:15 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 23:48:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 23:48:20 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 23:48:25 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 23:48:30 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 23:48:35 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 23:48:35 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/10 23:48:35 No eventing-con process running
2021/12/10 23:48:35 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/10 23:48:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:48:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:48:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:48:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:48:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (268.45s)
=== RUN   TestKVRebInWithTimers
2021/12/10 23:48:58 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 23:48:58 App: TestKVRebInWithTimers got un-deployed
2021/12/10 23:48:58 No eventing-con process running
2021/12/10 23:48:58 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/10 23:49:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:49:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:49:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:49:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:49:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:49:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:49:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:49:17 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 23:49:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 23:49:23 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 23:49:28 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 23:49:33 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 23:49:38 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 23:49:38 App: TestKVRebInWithTimers got deployed
2021/12/10 23:49:38 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 23:49:38 Starting bucket ops to source bucket
2021/12/10 23:49:38 Removing node(s): [] from the cluster
2021/12/10 23:49:43 Rebalance progress: 12.5
2021/12/10 23:49:48 Rebalance progress: 12.5
2021/12/10 23:49:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:49:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:49:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:49:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 23:49:58 Rebalance progress: 12.5
2021/12/10 23:50:03 Rebalance progress: 12.5
2021/12/10 23:50:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:50:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:50:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:50:09 Waiting for src bucket item count to get to: 50000 curr count: 27073
2021/12/10 23:50:14 Waiting for src bucket item count to get to: 50000 curr count: 37125
2021/12/10 23:50:19 Waiting for src bucket item count to get to: 50000 curr count: 37125
2021/12/10 23:50:24 Waiting for src bucket item count to get to: 50000 curr count: 49871
2021/12/10 23:50:29 Waiting for src bucket item count to get to: 50000 curr count: 49871
2021/12/10 23:50:39 Waiting for dst bucket item count to get to: 50000 curr count: 48697
2021/12/10 23:50:44 Waiting for dst bucket item count to get to: 50000 curr count: 48697
2021/12/10 23:50:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 23:50:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 23:50:44 Starting bucket ops to source bucket
2021/12/10 23:51:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:51:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:52:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:52:08 Waiting for dst bucket item count to get to: 0 curr count: 41043
2021/12/10 23:52:13 Waiting for dst bucket item count to get to: 0 curr count: 41043
2021/12/10 23:52:18 Waiting for dst bucket item count to get to: 0 curr count: 25512
2021/12/10 23:52:23 Waiting for dst bucket item count to get to: 0 curr count: 25512
2021/12/10 23:52:28 Waiting for dst bucket item count to get to: 0 curr count: 9463
2021/12/10 23:52:33 Waiting for dst bucket item count to get to: 0 curr count: 9463
2021/12/10 23:52:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:52:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 23:52:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 23:52:38 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 23:52:43 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 23:52:48 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 23:52:53 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 23:52:53 App: TestKVRebInWithTimers got un-deployed
2021/12/10 23:52:53 No eventing-con process running
2021/12/10 23:52:54 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/10 23:52:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:53:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:53:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:53:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:53:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:53:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (258.63s)
=== RUN   TestKVRebOutWithTimers
2021/12/10 23:53:12 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 23:53:12 Removing node(s): [] from the cluster
2021/12/10 23:53:17 Rebalance progress: 100
2021/12/10 23:53:22 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 23:53:22 App: TestKVRebOutWithTimers got un-deployed
2021/12/10 23:53:22 No eventing-con process running
2021/12/10 23:53:22 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/10 23:53:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:53:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:53:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:53:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:53:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:53:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:53:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:53:41 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 23:53:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 23:53:47 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 23:53:52 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 23:53:57 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 23:54:02 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 23:54:02 App: TestKVRebOutWithTimers got deployed
2021/12/10 23:54:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 23:54:02 Starting bucket ops to source bucket
2021/12/10 23:54:07 Rebalance progress: 12.5
2021/12/10 23:54:12 Rebalance progress: 12.5
2021/12/10 23:54:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:54:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:54:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:54:17 Waiting for src bucket item count to get to: 50000 curr count: 10005
2021/12/10 23:54:22 Waiting for src bucket item count to get to: 50000 curr count: 24407
2021/12/10 23:54:27 Waiting for src bucket item count to get to: 50000 curr count: 24407
2021/12/10 23:54:32 Waiting for src bucket item count to get to: 50000 curr count: 36198
2021/12/10 23:54:37 Waiting for src bucket item count to get to: 50000 curr count: 36198
2021/12/10 23:54:42 Waiting for src bucket item count to get to: 50000 curr count: 46030
2021/12/10 23:54:47 Waiting for src bucket item count to get to: 50000 curr count: 46030
2021/12/10 23:54:52 Waiting for src bucket item count to get to: 50000 curr count: 46030
2021/12/10 23:55:02 Waiting for dst bucket item count to get to: 50000 curr count: 36098
2021/12/10 23:55:07 Waiting for dst bucket item count to get to: 50000 curr count: 36098
2021/12/10 23:55:07 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 23:55:07 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 23:55:07 Starting bucket ops to source bucket
2021/12/10 23:56:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:56:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:56:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:56:31 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:56:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 23:56:41 Waiting for dst bucket item count to get to: 0 curr count: 35917
2021/12/10 23:56:46 Waiting for dst bucket item count to get to: 0 curr count: 35917
2021/12/10 23:56:51 Waiting for dst bucket item count to get to: 0 curr count: 20529
2021/12/10 23:56:56 Waiting for dst bucket item count to get to: 0 curr count: 20529
2021/12/10 23:57:01 Waiting for dst bucket item count to get to: 0 curr count: 7832
2021/12/10 23:57:06 Waiting for dst bucket item count to get to: 0 curr count: 7832
2021/12/10 23:57:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:57:06 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 23:57:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 23:57:12 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 23:57:17 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 23:57:22 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 23:57:27 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 23:57:27 App: TestKVRebOutWithTimers got un-deployed
2021/12/10 23:57:27 No eventing-con process running
2021/12/10 23:57:27 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/10 23:57:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:57:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:57:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:57:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:57:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:57:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (273.14s)
=== RUN   TestKVSwapRebWithTimers
2021/12/10 23:57:50 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 23:57:50 App: TestKVSwapRebWithTimers got un-deployed
2021/12/10 23:57:50 No eventing-con process running
2021/12/10 23:57:50 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/10 23:57:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:57:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:57:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 23:57:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:58:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:58:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:58:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 23:58:09 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 23:58:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 23:58:15 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 23:58:20 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 23:58:25 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 23:58:30 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 23:58:30 App: TestKVSwapRebWithTimers got deployed
2021/12/10 23:58:30 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 23:58:30 Starting bucket ops to source bucket
2021/12/10 23:58:30 Removing node(s): [] from the cluster
2021/12/10 23:58:35 Rebalance progress: 12.5
2021/12/10 23:58:40 Rebalance progress: 12.5
2021/12/10 23:58:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:58:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:58:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:58:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 23:58:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 23:58:51 Rebalance progress: 12.5
2021/12/10 23:58:56 Rebalance progress: 12.5
2021/12/10 23:59:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:59:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:59:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:59:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 23:59:06 Rebalance progress: 12.5
2021/12/10 23:59:11 Rebalance progress: 12.5
2021/12/10 23:59:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:59:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 23:59:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 23:59:16 Waiting for src bucket item count to get to: 50000 curr count: 45419
2021/12/10 23:59:26 Waiting for dst bucket item count to get to: 50000 curr count: 43770
2021/12/10 23:59:31 Waiting for dst bucket item count to get to: 50000 curr count: 43770
2021/12/10 23:59:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 23:59:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 23:59:31 Starting bucket ops to source bucket
2021/12/11 00:00:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:00:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:00:51 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:00:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:01:01 Waiting for dst bucket item count to get to: 0 curr count: 40728
2021/12/11 00:01:06 Waiting for dst bucket item count to get to: 0 curr count: 40728
2021/12/11 00:01:11 Waiting for dst bucket item count to get to: 0 curr count: 17963
2021/12/11 00:01:16 Waiting for dst bucket item count to get to: 0 curr count: 17963
2021/12/11 00:01:21 Waiting for dst bucket item count to get to: 0 curr count: 5984
2021/12/11 00:01:26 Waiting for dst bucket item count to get to: 0 curr count: 5984
2021/12/11 00:01:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:01:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 00:01:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:01:31 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 00:01:36 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 00:01:41 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 00:01:46 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 00:01:51 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 00:01:56 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 00:01:56 App: TestKVSwapRebWithTimers got un-deployed
2021/12/11 00:01:56 No eventing-con process running
2021/12/11 00:01:56 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/11 00:01:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:02:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:02:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:02:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:02:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:02:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (269.59s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/11 00:02:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 00:02:20 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/11 00:02:20 No eventing-con process running
2021/12/11 00:02:20 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/11 00:02:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:02:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:02:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:02:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:02:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:02:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:02:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:02:38 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:02:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 00:02:44 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 00:02:49 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 00:02:54 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 00:02:59 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 00:02:59 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/11 00:02:59 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 00:02:59 Starting bucket ops to source bucket
2021/12/11 00:03:00 Removing node(s): [] from the cluster
2021/12/11 00:03:05 Rebalance progress: 12.5
2021/12/11 00:03:10 Rebalance progress: 12.5
2021/12/11 00:03:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:03:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:03:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:03:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:03:20 Rebalance progress: 12.5
2021/12/11 00:03:25 Rebalance progress: 12.5
2021/12/11 00:03:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:03:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:03:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:03:30 Waiting for src bucket item count to get to: 50000 curr count: 26047
2021/12/11 00:03:35 Waiting for src bucket item count to get to: 50000 curr count: 38396
2021/12/11 00:03:40 Waiting for src bucket item count to get to: 50000 curr count: 48351
2021/12/11 00:03:45 Waiting for src bucket item count to get to: 50000 curr count: 48351
2021/12/11 00:03:55 Waiting for dst bucket item count to get to: 50000 curr count: 34960
2021/12/11 00:04:00 Waiting for dst bucket item count to get to: 50000 curr count: 34960
2021/12/11 00:04:05 Waiting for dst bucket item count to get to: 50000 curr count: 43329
2021/12/11 00:04:10 Waiting for dst bucket item count to get to: 50000 curr count: 43329
2021/12/11 00:04:15 Waiting for dst bucket item count to get to: 50000 curr count: 43329
2021/12/11 00:04:15 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:04:15 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:04:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:04:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 00:04:25 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 00:04:30 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 00:04:35 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 00:04:40 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 00:04:45 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 00:04:45 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/11 00:04:45 No eventing-con process running
2021/12/11 00:04:45 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/11 00:04:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:04:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:04:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:04:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:04:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:05:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:05:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:05:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.05s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/11 00:05:14 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 00:05:14 Removing node(s): [] from the cluster
2021/12/11 00:05:19 Rebalance progress: 100
2021/12/11 00:05:24 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 00:05:24 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/11 00:05:24 No eventing-con process running
2021/12/11 00:05:24 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/11 00:05:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:05:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:05:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:05:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:05:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:05:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:05:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:05:43 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:05:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 00:05:49 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 00:05:54 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 00:05:59 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 00:06:04 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 00:06:04 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/11 00:06:04 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:06:04 Starting bucket ops to source bucket
2021/12/11 00:06:09 Rebalance progress: 12.5
2021/12/11 00:06:14 Rebalance progress: 12.5
2021/12/11 00:06:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:06:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:06:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:06:19 Waiting for src bucket item count to get to: 50000 curr count: 6565
2021/12/11 00:06:24 Waiting for src bucket item count to get to: 50000 curr count: 20378
2021/12/11 00:06:29 Waiting for src bucket item count to get to: 50000 curr count: 20378
2021/12/11 00:06:34 Waiting for src bucket item count to get to: 50000 curr count: 32740
2021/12/11 00:06:39 Waiting for src bucket item count to get to: 50000 curr count: 32740
2021/12/11 00:06:44 Waiting for src bucket item count to get to: 50000 curr count: 43476
2021/12/11 00:06:49 Waiting for src bucket item count to get to: 50000 curr count: 43476
2021/12/11 00:06:54 Waiting for src bucket item count to get to: 50000 curr count: 43476
2021/12/11 00:07:04 Waiting for dst bucket item count to get to: 50000 curr count: 34488
2021/12/11 00:07:09 Waiting for dst bucket item count to get to: 50000 curr count: 34488
2021/12/11 00:07:14 Waiting for dst bucket item count to get to: 50000 curr count: 42554
2021/12/11 00:07:19 Waiting for dst bucket item count to get to: 50000 curr count: 42554
2021/12/11 00:07:19 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:07:19 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:07:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:07:24 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 00:07:29 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 00:07:34 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 00:07:39 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 00:07:44 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 00:07:49 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 00:07:49 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/11 00:07:49 No eventing-con process running
2021/12/11 00:07:49 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/11 00:07:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:07:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:08:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:08:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:08:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:08:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:08:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:08:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:08:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.13s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/11 00:08:28 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 00:08:28 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/11 00:08:28 No eventing-con process running
2021/12/11 00:08:28 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/11 00:08:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:08:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:08:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:08:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:08:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:08:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:08:47 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:08:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 00:08:53 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 00:08:58 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 00:09:03 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 00:09:08 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 00:09:08 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/11 00:09:08 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 00:09:08 Starting bucket ops to source bucket
2021/12/11 00:09:08 Removing node(s): [] from the cluster
2021/12/11 00:09:13 Rebalance progress: 12.5
2021/12/11 00:09:18 Rebalance progress: 12.5
2021/12/11 00:09:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:09:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:09:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:09:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 00:09:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:09:29 Rebalance progress: 12.5
2021/12/11 00:09:34 Rebalance progress: 12.5
2021/12/11 00:09:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:09:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:09:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:09:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 00:09:44 Rebalance progress: 12.5
2021/12/11 00:09:49 Rebalance progress: 12.5
2021/12/11 00:09:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:09:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:09:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:09:54 Waiting for src bucket item count to get to: 50000 curr count: 49182
2021/12/11 00:09:59 Waiting for src bucket item count to get to: 50000 curr count: 49182
2021/12/11 00:10:09 Waiting for dst bucket item count to get to: 50000 curr count: 38173
2021/12/11 00:10:14 Waiting for dst bucket item count to get to: 50000 curr count: 38173
2021/12/11 00:10:19 Waiting for dst bucket item count to get to: 50000 curr count: 45282
2021/12/11 00:10:24 Waiting for dst bucket item count to get to: 50000 curr count: 45282
2021/12/11 00:10:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:10:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:10:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:10:29 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 00:10:34 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 00:10:39 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 00:10:44 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 00:10:49 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 00:10:54 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 00:10:54 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/11 00:10:54 No eventing-con process running
2021/12/11 00:10:54 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/11 00:10:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:11:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:11:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:11:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:11:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:11:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:11:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:11:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.64s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/11 00:11:28 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 00:11:28 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/11 00:11:28 No eventing-con process running
2021/12/11 00:11:28 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/11 00:11:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:11:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:11:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:11:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:11:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:11:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:11:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:11:46 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:11:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 00:11:52 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 00:11:57 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 00:12:02 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 00:12:07 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 00:12:07 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/11 00:12:07 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:12:07 Starting bucket ops to source bucket
2021/12/11 00:12:08 Removing node(s): [] from the cluster
2021/12/11 00:12:13 Rebalance progress: 12.5
2021/12/11 00:12:18 Rebalance progress: 12.5
2021/12/11 00:12:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:12:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:12:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:12:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:12:28 Rebalance progress: 12.5
2021/12/11 00:12:33 Rebalance progress: 12.5
2021/12/11 00:12:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:12:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:12:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:12:38 Waiting for src bucket item count to get to: 50000 curr count: 28100
2021/12/11 00:12:43 Waiting for src bucket item count to get to: 50000 curr count: 38436
2021/12/11 00:12:48 Waiting for src bucket item count to get to: 50000 curr count: 38436
2021/12/11 00:12:53 Waiting for src bucket item count to get to: 50000 curr count: 49202
2021/12/11 00:12:58 Waiting for src bucket item count to get to: 50000 curr count: 49202
2021/12/11 00:13:08 Waiting for dst bucket item count to get to: 50000 curr count: 39223
2021/12/11 00:13:13 Waiting for dst bucket item count to get to: 50000 curr count: 39223
2021/12/11 00:13:18 Waiting for dst bucket item count to get to: 50000 curr count: 46173
2021/12/11 00:13:23 Waiting for dst bucket item count to get to: 50000 curr count: 46173
2021/12/11 00:13:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:13:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:13:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:13:28 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 00:13:33 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 00:13:38 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 00:13:43 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 00:13:48 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 00:13:53 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 00:13:53 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/11 00:13:53 No eventing-con process running
2021/12/11 00:13:53 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/11 00:13:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:14:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:14:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:14:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:14:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:14:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:14:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:14:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.24s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/11 00:14:22 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:14:22 Removing node(s): [] from the cluster
2021/12/11 00:14:27 Rebalance progress: 100
2021/12/11 00:14:32 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 00:14:32 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/11 00:14:32 No eventing-con process running
2021/12/11 00:14:32 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/11 00:14:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:14:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:14:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:14:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:14:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:14:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:14:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:14:51 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:14:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 00:14:57 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 00:15:02 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 00:15:07 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 00:15:12 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 00:15:12 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/11 00:15:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:15:12 Starting bucket ops to source bucket
2021/12/11 00:15:17 Rebalance progress: 12.5
2021/12/11 00:15:22 Rebalance progress: 12.5
2021/12/11 00:15:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:15:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:15:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:15:27 Waiting for src bucket item count to get to: 50000 curr count: 9350
2021/12/11 00:15:32 Waiting for src bucket item count to get to: 50000 curr count: 23113
2021/12/11 00:15:37 Waiting for src bucket item count to get to: 50000 curr count: 23113
2021/12/11 00:15:42 Waiting for src bucket item count to get to: 50000 curr count: 35870
2021/12/11 00:15:47 Waiting for src bucket item count to get to: 50000 curr count: 35870
2021/12/11 00:15:52 Waiting for src bucket item count to get to: 50000 curr count: 45300
2021/12/11 00:15:57 Waiting for src bucket item count to get to: 50000 curr count: 45300
2021/12/11 00:16:07 Waiting for dst bucket item count to get to: 50000 curr count: 32010
2021/12/11 00:16:12 Waiting for dst bucket item count to get to: 50000 curr count: 32010
2021/12/11 00:16:17 Waiting for dst bucket item count to get to: 50000 curr count: 32010
2021/12/11 00:16:22 Waiting for dst bucket item count to get to: 50000 curr count: 46506
2021/12/11 00:16:27 Waiting for dst bucket item count to get to: 50000 curr count: 46506
2021/12/11 00:16:27 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:16:27 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:16:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:16:32 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 00:16:37 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 00:16:42 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 00:16:47 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 00:16:52 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 00:16:57 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 00:16:57 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/11 00:16:57 No eventing-con process running
2021/12/11 00:16:57 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/11 00:16:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:17:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:17:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:17:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:17:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:17:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:17:21 Waiting for dst bucket item count to get to: 0 curr count: 49236
2021/12/11 00:17:26 Waiting for dst bucket item count to get to: 0 curr count: 49236
2021/12/11 00:17:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.18s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/11 00:17:36 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 00:17:36 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/11 00:17:36 No eventing-con process running
2021/12/11 00:17:36 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/11 00:17:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:17:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:17:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:17:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:17:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:17:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:17:55 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:17:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 00:18:01 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 00:18:06 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 00:18:11 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 00:18:16 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 00:18:16 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/11 00:18:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:18:16 Starting bucket ops to source bucket
2021/12/11 00:18:16 Removing node(s): [] from the cluster
2021/12/11 00:18:21 Rebalance progress: 12.5
2021/12/11 00:18:26 Rebalance progress: 12.5
2021/12/11 00:18:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:18:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:18:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:18:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 00:18:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:18:37 Rebalance progress: 12.5
2021/12/11 00:18:42 Rebalance progress: 12.5
2021/12/11 00:18:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:18:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:18:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:18:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 00:18:52 Rebalance progress: 12.5
2021/12/11 00:18:57 Rebalance progress: 12.5
2021/12/11 00:19:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:19:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:19:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:19:07 Waiting for dst bucket item count to get to: 50000 curr count: 28572
2021/12/11 00:19:12 Waiting for dst bucket item count to get to: 50000 curr count: 28572
2021/12/11 00:19:17 Waiting for dst bucket item count to get to: 50000 curr count: 35710
2021/12/11 00:19:22 Waiting for dst bucket item count to get to: 50000 curr count: 35710
2021/12/11 00:19:27 Waiting for dst bucket item count to get to: 50000 curr count: 46884
2021/12/11 00:19:32 Waiting for dst bucket item count to get to: 50000 curr count: 46884
2021/12/11 00:19:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:19:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:19:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:19:37 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 00:19:42 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 00:19:47 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 00:19:52 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 00:19:57 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 00:20:02 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 00:20:02 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/11 00:20:02 No eventing-con process running
2021/12/11 00:20:02 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/11 00:20:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:20:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:20:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:20:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:20:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:20:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:20:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:20:31 Waiting for dst bucket item count to get to: 0 curr count: 39674
2021/12/11 00:20:36 Waiting for dst bucket item count to get to: 0 curr count: 39674
2021/12/11 00:20:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (189.80s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/11 00:20:46 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 00:20:46 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/11 00:20:46 No eventing-con process running
2021/12/11 00:20:46 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/11 00:20:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:20:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:20:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:20:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:20:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:20:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:21:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:21:04 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:21:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 00:21:10 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 00:21:15 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 00:21:20 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 00:21:25 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 00:21:25 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/11 00:21:25 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 00:21:25 Starting bucket ops to source bucket
2021/12/11 00:21:26 Removing node(s): [] from the cluster
2021/12/11 00:21:31 Rebalance progress: 12.5
2021/12/11 00:21:36 Rebalance progress: 12.5
2021/12/11 00:21:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:21:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:21:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:21:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:21:46 Rebalance progress: 12.5
2021/12/11 00:21:51 Rebalance progress: 12.5
2021/12/11 00:21:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:21:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:21:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:21:56 Waiting for src bucket item count to get to: 50000 curr count: 28607
2021/12/11 00:22:01 Waiting for src bucket item count to get to: 50000 curr count: 40473
2021/12/11 00:22:06 Waiting for src bucket item count to get to: 50000 curr count: 40473
2021/12/11 00:22:16 Waiting for dst bucket item count to get to: 50000 curr count: 33438
2021/12/11 00:22:21 Waiting for dst bucket item count to get to: 50000 curr count: 33438
2021/12/11 00:22:26 Waiting for dst bucket item count to get to: 50000 curr count: 43346
2021/12/11 00:22:31 Waiting for dst bucket item count to get to: 50000 curr count: 43346
2021/12/11 00:22:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:22:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:22:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:22:36 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 00:22:41 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 00:22:46 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 00:22:51 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 00:22:56 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 00:23:01 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 00:23:01 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/11 00:23:01 No eventing-con process running
2021/12/11 00:23:01 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/11 00:23:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:23:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:23:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:23:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:23:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:23:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:23:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:23:30 Waiting for dst bucket item count to get to: 0 curr count: 97
2021/12/11 00:23:35 Waiting for dst bucket item count to get to: 0 curr count: 97
2021/12/11 00:23:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (179.10s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/11 00:23:40 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 00:23:40 Removing node(s): [] from the cluster
2021/12/11 00:23:46 Rebalance progress: 100
2021/12/11 00:23:51 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 00:23:51 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/11 00:23:51 No eventing-con process running
2021/12/11 00:23:51 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/11 00:23:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:23:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:23:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:23:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:24:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:24:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:24:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:24:09 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:24:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 00:24:15 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 00:24:20 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 00:24:25 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 00:24:30 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 00:24:30 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/11 00:24:30 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:24:30 Starting bucket ops to source bucket
2021/12/11 00:24:35 Rebalance progress: 12.5
2021/12/11 00:24:40 Rebalance progress: 12.5
2021/12/11 00:24:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:24:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:24:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:24:45 Waiting for src bucket item count to get to: 50000 curr count: 12915
2021/12/11 00:24:50 Waiting for src bucket item count to get to: 50000 curr count: 25195
2021/12/11 00:24:55 Waiting for src bucket item count to get to: 50000 curr count: 25195
2021/12/11 00:25:00 Waiting for src bucket item count to get to: 50000 curr count: 35378
2021/12/11 00:25:05 Waiting for src bucket item count to get to: 50000 curr count: 35378
2021/12/11 00:25:10 Waiting for src bucket item count to get to: 50000 curr count: 35378
2021/12/11 00:25:15 Waiting for src bucket item count to get to: 50000 curr count: 47499
2021/12/11 00:25:20 Waiting for src bucket item count to get to: 50000 curr count: 47499
2021/12/11 00:25:30 Waiting for dst bucket item count to get to: 50000 curr count: 45680
2021/12/11 00:25:35 Waiting for dst bucket item count to get to: 50000 curr count: 45680
2021/12/11 00:25:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:25:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:25:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:25:40 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 00:25:46 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 00:25:51 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 00:25:56 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 00:26:01 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 00:26:06 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 00:26:06 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/11 00:26:06 No eventing-con process running
2021/12/11 00:26:06 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/11 00:26:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:26:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:26:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:26:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:26:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:26:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:26:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (169.43s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/11 00:26:34 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 00:26:34 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/11 00:26:34 No eventing-con process running
2021/12/11 00:26:34 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/11 00:26:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:26:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:26:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:26:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:26:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:26:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:26:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:26:53 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:26:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 00:26:59 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 00:27:04 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 00:27:09 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 00:27:14 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 00:27:14 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/11 00:27:14 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 00:27:14 Starting bucket ops to source bucket
2021/12/11 00:27:14 Removing node(s): [] from the cluster
2021/12/11 00:27:20 Rebalance progress: 12.5
2021/12/11 00:27:25 Rebalance progress: 12.5
2021/12/11 00:27:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:27:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:27:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:27:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 00:27:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:27:35 Rebalance progress: 12.5
2021/12/11 00:27:40 Rebalance progress: 12.5
2021/12/11 00:27:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:27:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:27:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:27:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 00:27:50 Rebalance progress: 12.5
2021/12/11 00:27:55 Rebalance progress: 12.5
2021/12/11 00:28:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:28:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:28:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:28:00 Waiting for src bucket item count to get to: 50000 curr count: 40200
2021/12/11 00:28:10 Waiting for dst bucket item count to get to: 50000 curr count: 36167
2021/12/11 00:28:15 Waiting for dst bucket item count to get to: 50000 curr count: 36167
2021/12/11 00:28:20 Waiting for dst bucket item count to get to: 50000 curr count: 43909
2021/12/11 00:28:25 Waiting for dst bucket item count to get to: 50000 curr count: 43909
2021/12/11 00:28:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:28:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:28:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:28:30 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 00:28:35 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 00:28:41 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 00:28:46 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 00:28:51 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 00:28:56 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 00:28:56 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/11 00:28:56 No eventing-con process running
2021/12/11 00:28:56 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/11 00:28:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:29:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:29:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:29:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:29:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:29:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:29:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:29:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (174.82s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/11 00:29:29 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 00:29:29 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/11 00:29:29 No eventing-con process running
2021/12/11 00:29:29 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/11 00:29:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:29:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:29:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:29:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:29:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:29:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:29:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:29:48 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:29:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 00:29:54 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 00:29:59 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 00:30:04 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 00:30:09 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 00:30:09 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/11 00:30:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:30:09 Starting bucket ops to source bucket
2021/12/11 00:30:09 Removing node(s): [] from the cluster
2021/12/11 00:30:14 Rebalance progress: 12.5
2021/12/11 00:30:19 Rebalance progress: 12.5
2021/12/11 00:30:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:30:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:30:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:30:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:30:30 Rebalance progress: 12.5
2021/12/11 00:30:35 Rebalance progress: 12.5
2021/12/11 00:30:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:30:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:30:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:30:40 Waiting for src bucket item count to get to: 50000 curr count: 25626
2021/12/11 00:30:45 Waiting for src bucket item count to get to: 50000 curr count: 34800
2021/12/11 00:30:50 Waiting for src bucket item count to get to: 50000 curr count: 34800
2021/12/11 00:30:55 Waiting for src bucket item count to get to: 50000 curr count: 47113
2021/12/11 00:31:00 Waiting for src bucket item count to get to: 50000 curr count: 47113
2021/12/11 00:31:10 Waiting for dst bucket item count to get to: 50000 curr count: 44353
2021/12/11 00:31:15 Waiting for dst bucket item count to get to: 50000 curr count: 44353
2021/12/11 00:31:15 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:31:15 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:31:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:31:20 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 00:31:25 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 00:31:30 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 00:31:35 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 00:31:40 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 00:31:45 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 00:31:45 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/11 00:31:45 No eventing-con process running
2021/12/11 00:31:45 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/11 00:31:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:31:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:31:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:31:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:31:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:32:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:32:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:32:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.28s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/11 00:32:13 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:32:14 Removing node(s): [] from the cluster
2021/12/11 00:32:19 Rebalance progress: 100
2021/12/11 00:32:24 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 00:32:24 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/11 00:32:24 No eventing-con process running
2021/12/11 00:32:24 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/11 00:32:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:32:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:32:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:32:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:32:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:32:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:32:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:32:42 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:32:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 00:32:48 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 00:32:53 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 00:32:58 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 00:33:03 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 00:33:03 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/11 00:33:03 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:33:03 Starting bucket ops to source bucket
2021/12/11 00:33:09 Rebalance progress: 12.5
2021/12/11 00:33:14 Rebalance progress: 12.5
2021/12/11 00:33:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:33:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:33:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:33:19 Waiting for src bucket item count to get to: 50000 curr count: 7493
2021/12/11 00:33:24 Waiting for src bucket item count to get to: 50000 curr count: 20365
2021/12/11 00:33:29 Waiting for src bucket item count to get to: 50000 curr count: 20365
2021/12/11 00:33:34 Waiting for src bucket item count to get to: 50000 curr count: 31778
2021/12/11 00:33:39 Waiting for src bucket item count to get to: 50000 curr count: 31778
2021/12/11 00:33:44 Waiting for src bucket item count to get to: 50000 curr count: 43065
2021/12/11 00:33:49 Waiting for src bucket item count to get to: 50000 curr count: 43065
2021/12/11 00:33:54 Waiting for src bucket item count to get to: 50000 curr count: 43065
2021/12/11 00:34:04 Waiting for dst bucket item count to get to: 50000 curr count: 38368
2021/12/11 00:34:09 Waiting for dst bucket item count to get to: 50000 curr count: 38368
2021/12/11 00:34:14 Waiting for dst bucket item count to get to: 50000 curr count: 48838
2021/12/11 00:34:19 Waiting for dst bucket item count to get to: 50000 curr count: 48838
2021/12/11 00:34:19 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:34:19 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:34:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:34:24 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 00:34:29 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 00:34:34 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 00:34:39 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 00:34:44 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 00:34:44 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/11 00:34:44 No eventing-con process running
2021/12/11 00:34:44 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/11 00:34:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:34:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:34:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:34:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:34:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:35:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:35:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:35:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.37s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/11 00:35:18 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 00:35:18 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/11 00:35:18 No eventing-con process running
2021/12/11 00:35:18 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/11 00:35:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:35:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:35:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:35:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:35:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:35:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:35:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:35:36 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:35:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 00:35:42 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 00:35:47 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 00:35:52 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 00:35:57 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 00:35:57 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/11 00:35:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:35:57 Starting bucket ops to source bucket
2021/12/11 00:35:58 Removing node(s): [] from the cluster
2021/12/11 00:36:03 Rebalance progress: 12.5
2021/12/11 00:36:08 Rebalance progress: 12.5
2021/12/11 00:36:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:36:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:36:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:36:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 00:36:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:36:18 Rebalance progress: 12.5
2021/12/11 00:36:23 Rebalance progress: 12.5
2021/12/11 00:36:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:36:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:36:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:36:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 00:36:34 Rebalance progress: 12.5
2021/12/11 00:36:39 Rebalance progress: 12.5
2021/12/11 00:36:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:36:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:36:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:36:44 Waiting for src bucket item count to get to: 50000 curr count: 48348
2021/12/11 00:36:49 Waiting for src bucket item count to get to: 50000 curr count: 48348
2021/12/11 00:36:59 Waiting for dst bucket item count to get to: 50000 curr count: 46148
2021/12/11 00:37:04 Waiting for dst bucket item count to get to: 50000 curr count: 46148
2021/12/11 00:37:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 00:37:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 00:37:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:37:09 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 00:37:14 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 00:37:19 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 00:37:24 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 00:37:29 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 00:37:34 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 00:37:34 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/11 00:37:34 No eventing-con process running
2021/12/11 00:37:34 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/11 00:37:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:37:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:37:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:37:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:37:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:37:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:37:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 00:37:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (169.69s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/11 00:38:07 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:38:08 App: bucket_op_on_update got un-deployed
2021/12/11 00:38:08 No eventing-con process running
2021/12/11 00:38:08 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 00:38:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:38:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:38:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:38:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:38:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:38:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:38:26 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:38:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 00:38:32 Starting bucket ops to source bucket
2021/12/11 00:38:37 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:38:42 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:38:47 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:38:47 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:38:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:38:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:38:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:38:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 00:38:53 Rebalance progress: 12.5
2021/12/11 00:38:58 Rebalance progress: 12.5
2021/12/11 00:39:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:39:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:39:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:39:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 00:39:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 00:39:09 Rebalance progress: 12.5
2021/12/11 00:39:14 Rebalance progress: 12.5
2021/12/11 00:39:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:39:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:39:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:39:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 00:39:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 00:39:24 Rebalance progress: 12.5
2021/12/11 00:39:29 Rebalance progress: 12.5
2021/12/11 00:39:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:39:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:39:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:39:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:39:39 Rebalance progress: 12.5
2021/12/11 00:39:44 Rebalance progress: 12.5
2021/12/11 00:39:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:39:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:39:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:39:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 00:39:54 Rebalance progress: 12.5
2021/12/11 00:39:59 Rebalance progress: 12.5
2021/12/11 00:40:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:40:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:40:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:40:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 00:40:09 Rebalance progress: 12.5
2021/12/11 00:40:14 Rebalance progress: 12.5
2021/12/11 00:40:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:40:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:40:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:40:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:40:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:40:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:40:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:40:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:40:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:40:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:40:50 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:40:50 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:40:50 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 00:40:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:40:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:41:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10710
2021/12/11 00:41:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10710
2021/12/11 00:41:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:41:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:41:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:41:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10710
2021/12/11 00:41:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.64s)
=== RUN   TestKVForceFailover
2021/12/11 00:41:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:41:33 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:41:33 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:41:33 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 00:41:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:41:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:41:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:41:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:41:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:41:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:41:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:41:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 00:41:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:41:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:41:52 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:41:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 00:41:53 Starting bucket ops to source bucket
2021/12/11 00:41:58 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:42:03 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:42:08 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:42:13 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:42:13 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:42:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:42:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:42:13 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/11 00:42:33 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/11 00:42:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 00:42:38 Rebalance progress: 12.5
2021/12/11 00:42:43 Rebalance progress: 12.5
2021/12/11 00:42:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:42:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:42:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:42:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/11 00:42:54 Rebalance progress: 12.5
2021/12/11 00:42:59 Rebalance progress: 12.5
2021/12/11 00:43:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:43:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:43:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:43:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:43:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:43:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:43:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:43:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:43:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:43:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:43:34 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:43:34 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:43:34 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 00:43:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:43:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:43:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7042
2021/12/11 00:43:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:43:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:43:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:43:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7042
2021/12/11 00:43:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.31s)
=== RUN   TestKVHardFailoverRecovery
2021/12/11 00:44:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:44:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:44:07 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:44:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 00:44:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:44:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:44:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:44:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:44:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:44:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:44:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:44:26 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:44:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 00:44:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 00:44:37 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:44:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:44:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:44:47 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:44:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:44:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:44:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:44:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 00:44:53 Rebalance progress: 12.5
2021/12/11 00:44:58 Rebalance progress: 12.5
2021/12/11 00:45:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:45:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:45:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:45:03 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/11 00:46:03 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/11 00:46:08 Rebalance progress: 12.5
2021/12/11 00:46:13 Rebalance progress: 12.5
2021/12/11 00:46:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:46:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:46:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:46:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:46:23 Rebalance progress: 12.5
2021/12/11 00:46:28 Rebalance progress: 12.5
2021/12/11 00:46:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:46:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:46:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:46:33 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:46:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:46:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:46:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:46:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:46:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:47:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:47:03 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:47:03 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:47:03 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 00:47:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:47:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:47:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12063
2021/12/11 00:47:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:47:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:47:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:47:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12063
2021/12/11 00:47:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.36s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/11 00:47:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:47:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:47:37 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:47:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 00:47:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:47:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:47:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:47:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:47:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:47:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:47:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:47:55 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:47:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 00:48:02 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 00:48:07 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:48:12 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:48:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:48:17 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:48:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:48:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:48:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:48:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 00:48:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 00:48:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 00:48:23 Rebalance progress: 12.5
2021/12/11 00:48:28 Rebalance progress: 12.5
2021/12/11 00:48:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:48:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:48:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:48:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/11 00:48:38 Rebalance progress: 12.5
2021/12/11 00:48:43 Rebalance progress: 12.5
2021/12/11 00:48:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:48:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:48:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:48:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:49:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:49:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:49:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:49:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:49:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:49:18 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:49:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 00:49:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:49:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:49:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 19
2021/12/11 00:49:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 19
2021/12/11 00:49:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:49:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:49:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:49:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4624
2021/12/11 00:49:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (139.87s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/11 00:49:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:49:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:49:57 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:49:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 00:49:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:50:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:50:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:50:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:50:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:50:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:50:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:50:10 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:50:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 00:50:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 00:50:21 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:50:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:50:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:50:31 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:50:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:50:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:50:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:50:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 00:50:37 Rebalance progress: 12.5
2021/12/11 00:50:42 Rebalance progress: 12.5
2021/12/11 00:50:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:50:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:50:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:50:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 00:50:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:50:52 Rebalance progress: 12.5
2021/12/11 00:50:57 Rebalance progress: 12.5
2021/12/11 00:51:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:51:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:51:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:51:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 00:51:07 Rebalance progress: 12.5
2021/12/11 00:51:12 Rebalance progress: 12.5
2021/12/11 00:51:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:51:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:51:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:51:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:51:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:51:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:51:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:51:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:51:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:51:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:51:48 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:51:48 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:51:48 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 00:51:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:51:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:51:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6076
2021/12/11 00:51:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:52:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:52:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:52:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6076
2021/12/11 00:52:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6076
2021/12/11 00:52:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:52:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:52:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:52:21 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:52:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 00:52:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:52:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:52:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:52:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:52:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:52:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (167.93s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/11 00:52:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:52:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:52:45 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:52:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 00:52:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:52:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:52:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:52:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:52:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:52:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:53:03 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 00:53:03 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 00:53:03 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:53:03 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/11 00:53:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:53:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:53:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:53:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:53:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:53:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:53:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:53:17 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:53:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:53:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:53:18 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:53:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/11 00:53:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 00:53:29 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:53:34 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:53:39 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:53:39 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/11 00:53:44 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 00:53:49 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 00:53:54 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 00:53:54 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:53:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:53:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:53:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:53:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 00:54:00 Rebalance progress: 12.5
2021/12/11 00:54:05 Rebalance progress: 12.5
2021/12/11 00:54:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:54:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:54:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:54:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 00:54:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 00:54:15 Rebalance progress: 12.5
2021/12/11 00:54:20 Rebalance progress: 12.5
2021/12/11 00:54:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:54:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:54:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:54:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 00:54:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 00:54:30 Rebalance progress: 12.5
2021/12/11 00:54:35 Rebalance progress: 12.5
2021/12/11 00:54:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:54:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:54:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:54:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:54:46 Rebalance progress: 12.5
2021/12/11 00:54:51 Rebalance progress: 12.5
2021/12/11 00:54:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:54:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:54:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:54:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 00:55:01 Rebalance progress: 12.5
2021/12/11 00:55:06 Rebalance progress: 12.5
2021/12/11 00:55:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:55:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:55:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:55:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 00:55:16 Rebalance progress: 12.5
2021/12/11 00:55:21 Rebalance progress: 12.5
2021/12/11 00:55:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:55:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:55:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:55:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:55:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:55:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:55:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:55:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:55:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:55:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:56:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:56:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:56:06 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:56:06 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:56:06 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 00:56:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:56:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:56:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:56:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:56:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:56:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12185
2021/12/11 00:56:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:56:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:56:32 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 00:56:37 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 00:56:42 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 00:56:47 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 00:56:53 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 00:56:53 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 00:56:53 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:56:53 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/11 00:56:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:56:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:56:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:57:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:57:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:57:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (271.52s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/11 00:57:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:57:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:57:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:57:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 00:57:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:57:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:57:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:57:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:57:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:57:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:57:35 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 00:57:35 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 00:57:35 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:57:35 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/11 00:57:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:57:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:57:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:57:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:57:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:57:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:57:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:57:48 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:57:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:57:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:57:49 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 00:57:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/11 00:57:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 00:58:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:58:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:58:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 00:58:10 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/11 00:58:15 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 00:58:21 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 00:58:26 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 00:58:26 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:58:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:58:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:58:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 00:58:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 00:58:31 Rebalance progress: 12.5
2021/12/11 00:58:36 Rebalance progress: 12.5
2021/12/11 00:58:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:58:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:58:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:58:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 00:58:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 00:58:47 Rebalance progress: 12.5
2021/12/11 00:58:52 Rebalance progress: 12.5
2021/12/11 00:58:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:58:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:58:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:58:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 00:59:02 Rebalance progress: 12.5
2021/12/11 00:59:07 Rebalance progress: 12.5
2021/12/11 00:59:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:59:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:59:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 00:59:12 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 00:59:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:59:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:59:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:59:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:59:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:59:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 00:59:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 00:59:42 eventingSetup.go:46: No eventing-con process running
2021/12/11 00:59:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 00:59:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 00:59:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 00:59:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7602
2021/12/11 00:59:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 00:59:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:00:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:00:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7602
2021/12/11 01:00:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:00:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:00:12 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 01:00:17 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 01:00:22 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 01:00:27 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 01:00:33 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 01:00:38 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 01:00:38 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 01:00:38 eventingSetup.go:46: No eventing-con process running
2021/12/11 01:00:38 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/11 01:00:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:00:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:00:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:00:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:00:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:00:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (224.99s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/11 01:01:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:01:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 01:01:01 eventingSetup.go:46: No eventing-con process running
2021/12/11 01:01:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 01:01:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:01:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:01:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:01:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:01:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:01:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:01:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:01:20 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:01:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 01:01:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 01:01:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 01:01:36 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 01:01:41 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 01:01:41 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:01:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:01:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:01:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 01:01:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 01:02:01 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:02:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:02:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:02:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:02:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:02:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 01:02:17 Rebalance progress: 12.5
2021/12/11 01:02:22 Rebalance progress: 12.5
2021/12/11 01:02:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:02:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:02:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:02:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 01:02:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 01:02:32 Rebalance progress: 12.5
2021/12/11 01:02:37 Rebalance progress: 12.5
2021/12/11 01:02:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:02:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:02:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:02:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 01:02:47 Rebalance progress: 12.5
2021/12/11 01:02:52 Rebalance progress: 12.5
2021/12/11 01:02:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:02:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:02:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:02:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:03:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:03:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:03:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:03:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:03:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:03:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:03:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 01:03:28 eventingSetup.go:46: No eventing-con process running
2021/12/11 01:03:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 01:03:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:03:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:03:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:03:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:03:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:03:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9165
2021/12/11 01:03:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (175.12s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/11 01:03:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:03:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 01:03:56 eventingSetup.go:46: No eventing-con process running
2021/12/11 01:03:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 01:03:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:04:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:04:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:04:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:04:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:04:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:04:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:04:15 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:04:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 01:04:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 01:04:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 01:04:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 01:04:36 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 01:04:36 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:04:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:04:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:04:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 01:04:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 01:04:41 Rebalance progress: 12.5
2021/12/11 01:04:46 Rebalance progress: 12.5
2021/12/11 01:04:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:04:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:04:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:04:51 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:05:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:05:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:05:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 01:05:06 Rebalance progress: 12.5
2021/12/11 01:05:11 Rebalance progress: 12.5
2021/12/11 01:05:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:05:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 01:05:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 01:05:22 Rebalance progress: 12.5
2021/12/11 01:05:27 Rebalance progress: 12.5
2021/12/11 01:05:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:05:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:05:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:05:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 01:05:37 Rebalance progress: 12.5
2021/12/11 01:05:42 Rebalance progress: 12.5
2021/12/11 01:05:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:05:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:05:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:05:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:05:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:05:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:06:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:06:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:06:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:06:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:06:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 01:06:17 eventingSetup.go:46: No eventing-con process running
2021/12/11 01:06:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 01:06:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:06:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:06:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8595
2021/12/11 01:06:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:06:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:06:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:06:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7037
2021/12/11 01:06:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7037
2021/12/11 01:06:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.55s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/11 01:06:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 01:06:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 01:06:57 Rebalance progress: 100
2021/12/11 01:06:57 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:07:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:07:27 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 01:07:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 01:07:33 Removing node(s): [] from the cluster
2021/12/11 01:07:33 Starting bucket ops to source bucket
2021/12/11 01:07:38 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/11 01:07:43 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 01:07:48 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 01:07:48 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:07:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:07:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 01:07:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 01:07:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:07:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:08:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:08:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:08:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:08:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 01:08:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 01:08:18 eventingSetup.go:46: No eventing-con process running
2021/12/11 01:08:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 01:08:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:08:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:08:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 13
2021/12/11 01:08:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 13
2021/12/11 01:08:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 01:08:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 01:08:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 01:08:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1504
2021/12/11 01:08:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.86s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12036.505s

Rebalance tests took 3 hours, 20 minutes and 39 seconds.

Testing Succeeded


Version: versions-10.12.2021-17.32.cfg
Build Log: make-10.12.2021-17.32.log
Server Log: logs-10.12.2021-17.32.tar.gz

Finished