Building

Started building at 2021/12/02 11:23:06
Calculating base
Updating mirror
Basing run on 7.1.0-1824 5c2f64433
Updating tree for run 02.12.2021-11.23
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at a00b35f, changes since last good build: none
kv_engine is at d82da78, changes since last good build: none
ns_server is at b0d5d34, changes since last good build: 
 b0d5d34d4 Merge remote-tracking branch 'couchbase/mad-hatter'
 038b7b0b3 MB-49147:[BP] Fixes for slow DETS
couchstore is at 05c4320, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 9c2ee17, changes since last good build: 
 9c2ee1780 Additional N1QL UDF test cases
Switching eventing to unstable
eventing is at 509cf97, changes since last good build: 
 509cf978 MB-49148: Grant SystemXattrWrite privilege to on_behalf_of_ user
Switching eventing-ee to unstable
eventing-ee is at 7060ef8, changes since last good build: none
Pulling in uncommitted change 158037 at refs/changes/37/158037/4
Total 7 (delta 5), reused 7 (delta 5)
[unstable 22260ee2] MB-47385: Move eventing-producer and cbevent go version to 1.16.5
 Author: AnkitPrabhu 
 Date: Thu Jul 22 11:08:10 2021 +0530
 3 files changed, 23 insertions(+), 4 deletions(-)
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 45813521] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 7f7a7146] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable ae7c50ac] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable b9dcecf6] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/02 11:37:58
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/02 11:39:29 Started functional
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:41850->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:41850->127.0.0.1:9000: read: connection reset by peer
2021/12/02 11:39:39 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/02 11:39:45 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/02 11:39:45 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 11:39:50 Removing node(s): [] from the cluster
2021/12/02 11:39:55 Rebalance progress: 2.640335648148148
2021/12/02 11:40:00 Rebalance progress: 7.269965277777778
2021/12/02 11:40:05 Rebalance progress: 10.41666666666667
2021/12/02 11:40:10 Rebalance progress: 15.0462962962963
2021/12/02 11:40:16 Rebalance progress: 18.51851851851852
2021/12/02 11:40:20 Rebalance progress: 22.75028935185185
2021/12/02 11:40:25 Rebalance progress: 27.25332754629629
2021/12/02 11:40:30 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/02 11:40:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:40:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:41:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:41:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:41:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:41:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:42:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:42:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:42:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:42:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:43:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:43:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:43:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:43:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:44:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:44:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:44:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:44:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:45:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:45:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:45:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:45:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 11:46:06 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:46:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/02 11:46:12 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/02 11:46:17 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/02 11:46:22 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/02 11:46:27 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/02 11:46:27 App: TestAdvancedGetOps got deployed
2021/12/02 11:46:27 Starting bucket ops to source bucket
2021/12/02 11:46:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 11:46:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 11:46:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:46:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:46:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:46:37 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/02 11:46:42 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/02 11:46:47 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/02 11:46:52 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/02 11:46:52 App: TestAdvancedGetOps got un-deployed
2021/12/02 11:46:52 No eventing-con process running
2021/12/02 11:46:52 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/02 11:46:54 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:47:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 11:47:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 11:47:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:47:11 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:47:21 Waiting for dst bucket item count to get to: 0 curr count: 89
2021/12/02 11:47:26 Waiting for dst bucket item count to get to: 0 curr count: 89
2021/12/02 11:47:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (410.24s)
=== RUN   TestAdvancedInsertOps
2021/12/02 11:47:26 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:47:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/02 11:47:32 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 11:47:37 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 11:47:42 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 11:47:47 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 11:47:47 App: TestAdvancedInsertOps got deployed
2021/12/02 11:47:47 Starting bucket ops to source bucket
2021/12/02 11:47:53 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/02 11:47:58 Waiting for dst bucket item count to get to: 1024 curr count: 622
2021/12/02 11:48:03 Waiting for dst bucket item count to get to: 1024 curr count: 622
2021/12/02 11:48:03 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/02 11:48:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:48:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:48:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:48:08 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 11:48:13 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 11:48:18 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 11:48:23 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 11:48:23 App: TestAdvancedInsertOps got un-deployed
2021/12/02 11:48:23 No eventing-con process running
2021/12/02 11:48:23 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/02 11:48:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:48:35 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 11:48:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:48:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:48:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 11:48:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:48:47 Testing insert operation on source bucket
2021/12/02 11:48:47 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:48:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/02 11:48:53 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 11:48:58 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 11:49:03 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 11:49:08 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 11:49:08 App: TestAdvancedInsertOps got deployed
2021/12/02 11:49:08 Starting bucket ops to source bucket
2021/12/02 11:49:14 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/02 11:49:19 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/02 11:49:24 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/02 11:49:24 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/02 11:49:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:49:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:49:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:49:29 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 11:49:34 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 11:49:39 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 11:49:44 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 11:49:44 App: TestAdvancedInsertOps got un-deployed
2021/12/02 11:49:44 No eventing-con process running
2021/12/02 11:49:44 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/02 11:49:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:49:56 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/02 11:49:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:49:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:50:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.75s)
=== RUN   TestAdvancedUpsertOps
2021/12/02 11:50:03 Starting bucket ops to source bucket
2021/12/02 11:50:03 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:50:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/02 11:50:09 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 11:50:14 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 11:50:19 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 11:50:24 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 11:50:24 App: TestAdvancedUpsertOps got deployed
2021/12/02 11:50:24 Starting bucket ops to source bucket
2021/12/02 11:50:30 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 11:50:35 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/02 11:50:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:50:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:50:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:50:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:50:40 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 11:50:45 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 11:50:50 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 11:50:55 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 11:50:55 App: TestAdvancedUpsertOps got un-deployed
2021/12/02 11:50:56 No eventing-con process running
2021/12/02 11:50:56 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/02 11:50:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:51:07 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 11:51:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:51:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:51:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:51:14 Testing upsert operation on source bucket
2021/12/02 11:51:14 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:51:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/02 11:51:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 11:51:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 11:51:30 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 11:51:35 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 11:51:35 App: TestAdvancedUpsertOps got deployed
2021/12/02 11:51:35 Starting bucket ops to source bucket
2021/12/02 11:51:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:51:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:51:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:51:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:51:41 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 11:51:46 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 11:51:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 11:51:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 11:51:56 App: TestAdvancedUpsertOps got un-deployed
2021/12/02 11:51:57 No eventing-con process running
2021/12/02 11:51:57 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/02 11:51:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:52:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:52:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:52:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (128.09s)
=== RUN   TestAdvancedReplaceOps
2021/12/02 11:52:11 Starting bucket ops to source bucket
2021/12/02 11:52:12 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:52:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/02 11:52:18 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 11:52:23 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 11:52:28 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 11:52:33 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 11:52:33 App: TestAdvancedReplaceOps got deployed
2021/12/02 11:52:33 Starting bucket ops to source bucket
2021/12/02 11:52:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 11:52:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 11:52:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:52:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:52:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:52:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:52:49 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 11:52:54 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 11:52:59 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 11:53:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 11:53:04 App: TestAdvancedReplaceOps got un-deployed
2021/12/02 11:53:04 No eventing-con process running
2021/12/02 11:53:04 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/02 11:53:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:53:16 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 11:53:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:53:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:53:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:53:23 Testing upsert operation on source bucket
2021/12/02 11:53:23 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:53:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/02 11:53:29 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 11:53:34 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 11:53:39 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 11:53:44 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 11:53:44 App: TestAdvancedReplaceOps got deployed
2021/12/02 11:53:44 Starting bucket ops to source bucket
2021/12/02 11:53:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:53:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:53:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:53:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:53:50 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 11:53:55 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 11:54:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 11:54:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 11:54:05 App: TestAdvancedReplaceOps got un-deployed
2021/12/02 11:54:05 No eventing-con process running
2021/12/02 11:54:05 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/02 11:54:07 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:54:17 Waiting for dst bucket item count to get to: 0 curr count: 157
2021/12/02 11:54:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:54:19 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:54:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (133.07s)
=== RUN   TestAdvancedDeleteOps
2021/12/02 11:54:24 Starting bucket ops to source bucket
2021/12/02 11:54:24 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:54:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/02 11:54:30 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 11:54:35 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 11:54:40 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 11:54:45 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 11:54:45 App: TestAdvancedDeleteOps got deployed
2021/12/02 11:54:45 Starting bucket ops to source bucket
2021/12/02 11:54:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 11:54:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 11:54:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:54:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:54:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:54:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:55:00 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 11:55:05 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 11:55:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 11:55:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 11:55:15 App: TestAdvancedDeleteOps got un-deployed
2021/12/02 11:55:15 No eventing-con process running
2021/12/02 11:55:15 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/02 11:55:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:55:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 11:55:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:55:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:55:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:55:34 Testing delete operation on source bucket
2021/12/02 11:55:34 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:55:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/02 11:55:40 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 11:55:45 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 11:55:50 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 11:55:55 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 11:55:55 App: TestAdvancedDeleteOps got deployed
2021/12/02 11:55:55 Starting bucket ops to source bucket
2021/12/02 11:55:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:55:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:55:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:55:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:56:00 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 11:56:05 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 11:56:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 11:56:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 11:56:15 App: TestAdvancedDeleteOps got un-deployed
2021/12/02 11:56:15 No eventing-con process running
2021/12/02 11:56:15 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/02 11:56:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:56:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:56:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:56:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.90s)
=== RUN   TestEnoentAdvancedGet
2021/12/02 11:56:30 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:56:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/02 11:56:36 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 11:56:41 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 11:56:46 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 11:56:51 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 11:56:51 App: TestEnoentAdvancedGet got deployed
2021/12/02 11:56:51 Starting bucket ops to source bucket
2021/12/02 11:56:56 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 11:57:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 11:57:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 11:57:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:57:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:57:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:57:06 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 11:57:11 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 11:57:16 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 11:57:21 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 11:57:21 App: TestEnoentAdvancedGet got un-deployed
2021/12/02 11:57:21 No eventing-con process running
2021/12/02 11:57:21 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/02 11:57:23 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:57:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 11:57:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 11:57:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:57:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:57:50 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 11:57:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 11:57:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:57:55 Testing get enoent operation on source bucket
2021/12/02 11:57:55 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:57:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/02 11:58:01 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 11:58:06 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 11:58:11 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 11:58:16 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 11:58:16 App: TestEnoentAdvancedGet got deployed
2021/12/02 11:58:16 Starting bucket ops to source bucket
2021/12/02 11:58:21 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/02 11:58:21 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/02 11:58:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:58:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:58:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:58:26 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 11:58:31 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 11:58:36 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 11:58:41 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 11:58:41 App: TestEnoentAdvancedGet got un-deployed
2021/12/02 11:58:41 No eventing-con process running
2021/12/02 11:58:41 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/02 11:58:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:58:53 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/02 11:58:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 11:58:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:59:00 Context: Bucket flush, response code: 200 dump: 
2021/12/02 11:59:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.03s)
=== RUN   TestEnoentAdvancedReplace
2021/12/02 11:59:05 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:59:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 11:59:11 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 11:59:16 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 11:59:21 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 11:59:26 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 11:59:26 App: TestEnoentAdvancedReplace got deployed
2021/12/02 11:59:26 Starting bucket ops to source bucket
2021/12/02 11:59:31 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 11:59:31 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 11:59:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 11:59:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 11:59:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:59:36 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 11:59:41 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 11:59:46 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 11:59:51 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 11:59:51 App: TestEnoentAdvancedReplace got un-deployed
2021/12/02 11:59:51 No eventing-con process running
2021/12/02 11:59:51 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/02 11:59:53 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:00:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:00:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:00:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:00:10 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:00:20 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:00:25 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:00:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:00:25 Testing get enoent operation on source bucket
2021/12/02 12:00:25 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:00:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 12:00:31 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 12:00:36 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 12:00:41 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 12:00:46 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 12:00:46 App: TestEnoentAdvancedReplace got deployed
2021/12/02 12:00:46 Starting bucket ops to source bucket
2021/12/02 12:00:51 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/02 12:00:51 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/02 12:00:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:00:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:00:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:00:56 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 12:01:01 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 12:01:06 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 12:01:11 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 12:01:11 App: TestEnoentAdvancedReplace got un-deployed
2021/12/02 12:01:11 No eventing-con process running
2021/12/02 12:01:11 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/02 12:01:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:01:23 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/02 12:01:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:01:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:01:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:01:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.29s)
=== RUN   TestEnoentAdvancedDelete
2021/12/02 12:01:35 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:01:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 12:01:41 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 12:01:46 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 12:01:51 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 12:01:56 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 12:01:56 App: TestEnoentAdvancedDelete got deployed
2021/12/02 12:01:56 Starting bucket ops to source bucket
2021/12/02 12:02:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 12:02:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 12:02:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:02:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:02:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:02:06 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 12:02:11 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 12:02:16 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 12:02:21 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 12:02:21 App: TestEnoentAdvancedDelete got un-deployed
2021/12/02 12:02:21 No eventing-con process running
2021/12/02 12:02:21 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/02 12:02:23 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:02:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:02:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:02:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:02:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:02:50 Waiting for dst bucket item count to get to: 0 curr count: 63
2021/12/02 12:02:55 Waiting for dst bucket item count to get to: 0 curr count: 63
2021/12/02 12:02:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:02:55 Testing get enoent operation on source bucket
2021/12/02 12:02:55 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:02:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 12:03:01 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 12:03:06 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 12:03:11 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 12:03:16 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 12:03:16 App: TestEnoentAdvancedDelete got deployed
2021/12/02 12:03:16 Starting bucket ops to source bucket
2021/12/02 12:03:21 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/02 12:03:21 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/02 12:03:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:03:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:03:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:03:27 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 12:03:32 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 12:03:37 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 12:03:42 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 12:03:42 App: TestEnoentAdvancedDelete got un-deployed
2021/12/02 12:03:42 No eventing-con process running
2021/12/02 12:03:42 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/02 12:03:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:03:53 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/02 12:03:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:03:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:04:00 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:04:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.29s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/02 12:04:05 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:04:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/02 12:04:11 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 12:04:16 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 12:04:21 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 12:04:26 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 12:04:26 App: TestAdvancedInsertKeyExist got deployed
2021/12/02 12:04:26 Starting bucket ops to source bucket
2021/12/02 12:04:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:04:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:04:27 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:04:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:04:32 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 12:04:37 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 12:04:42 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 12:04:47 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 12:04:47 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/02 12:04:47 No eventing-con process running
2021/12/02 12:04:47 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/02 12:04:48 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:04:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:04:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:05:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:05:01 Pumping document in the destination bucket
2021/12/02 12:05:01 Starting bucket ops to source bucket
2021/12/02 12:05:01 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:05:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/02 12:05:07 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 12:05:12 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 12:05:17 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 12:05:22 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 12:05:22 App: TestAdvancedInsertKeyExist got deployed
2021/12/02 12:05:22 Starting bucket ops to source bucket
2021/12/02 12:05:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:05:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:05:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:05:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:05:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:05:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:05:38 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 12:05:43 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 12:05:48 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 12:05:53 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 12:05:53 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/02 12:05:53 No eventing-con process running
2021/12/02 12:05:53 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/02 12:05:54 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:06:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:06:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:06:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:06:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (126.01s)
=== RUN   TestExipryGet
2021/12/02 12:06:11 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:06:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/02 12:06:17 Waiting for app: TestExipryGet to get deployed
2021/12/02 12:06:22 Waiting for app: TestExipryGet to get deployed
2021/12/02 12:06:27 Waiting for app: TestExipryGet to get deployed
2021/12/02 12:06:32 Waiting for app: TestExipryGet to get deployed
2021/12/02 12:06:32 App: TestExipryGet got deployed
2021/12/02 12:06:32 Starting bucket ops to source bucket
2021/12/02 12:07:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:07:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:07:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:07:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:07:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:07:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:07:28 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 12:07:33 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 12:07:38 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 12:07:43 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 12:07:43 App: TestExipryGet got un-deployed
2021/12/02 12:07:43 No eventing-con process running
2021/12/02 12:07:43 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/02 12:07:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:07:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:07:52 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:07:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:07:57 Pumping document in the destination bucket for expiry get
2021/12/02 12:07:57 Starting bucket ops to source bucket
2021/12/02 12:07:57 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:07:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/02 12:08:03 Waiting for app: TestExipryGet to get deployed
2021/12/02 12:08:09 Waiting for app: TestExipryGet to get deployed
2021/12/02 12:08:14 Waiting for app: TestExipryGet to get deployed
2021/12/02 12:08:19 Waiting for app: TestExipryGet to get deployed
2021/12/02 12:08:19 App: TestExipryGet got deployed
2021/12/02 12:08:19 Starting bucket ops to source bucket
2021/12/02 12:09:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:09:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:09:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:09:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:09:09 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:09:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:09:14 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 12:09:19 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 12:09:24 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 12:09:29 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 12:09:29 App: TestExipryGet got un-deployed
2021/12/02 12:09:29 No eventing-con process running
2021/12/02 12:09:29 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/02 12:09:31 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:09:41 Waiting for dst bucket item count to get to: 0 curr count: 84
2021/12/02 12:09:46 Waiting for dst bucket item count to get to: 0 curr count: 84
2021/12/02 12:09:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:09:47 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:09:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (221.21s)
=== RUN   TestExipryInsert
2021/12/02 12:09:52 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:09:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/02 12:09:59 Waiting for app: TestExipryInsert to get deployed
2021/12/02 12:10:04 Waiting for app: TestExipryInsert to get deployed
2021/12/02 12:10:09 Waiting for app: TestExipryInsert to get deployed
2021/12/02 12:10:14 Waiting for app: TestExipryInsert to get deployed
2021/12/02 12:10:14 App: TestExipryInsert got deployed
2021/12/02 12:10:14 Starting bucket ops to source bucket
2021/12/02 12:10:59 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/02 12:11:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:11:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:11:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:11:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:11:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:11:09 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 12:11:14 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 12:11:19 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 12:11:24 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 12:11:24 App: TestExipryInsert got un-deployed
2021/12/02 12:11:24 No eventing-con process running
2021/12/02 12:11:24 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/02 12:11:26 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:11:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:11:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:11:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:11:38 Testing on the destination bucket for expiry insert
2021/12/02 12:11:38 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:11:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/02 12:11:44 Waiting for app: TestExipryInsert to get deployed
2021/12/02 12:11:49 Waiting for app: TestExipryInsert to get deployed
2021/12/02 12:11:54 Waiting for app: TestExipryInsert to get deployed
2021/12/02 12:11:59 Waiting for app: TestExipryInsert to get deployed
2021/12/02 12:11:59 App: TestExipryInsert got deployed
2021/12/02 12:11:59 Starting bucket ops to source bucket
2021/12/02 12:12:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:12:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:12:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:12:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:12:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:12:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:12:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:13:00 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 12:13:05 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 12:13:10 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 12:13:15 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 12:13:15 App: TestExipryInsert got un-deployed
2021/12/02 12:13:15 No eventing-con process running
2021/12/02 12:13:15 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/02 12:13:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:13:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:13:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:13:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:13:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.98s)
=== RUN   TestExipryReplace
2021/12/02 12:13:33 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:13:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 12:13:40 Waiting for app: TestExipryReplace to get deployed
2021/12/02 12:13:45 Waiting for app: TestExipryReplace to get deployed
2021/12/02 12:13:50 Waiting for app: TestExipryReplace to get deployed
2021/12/02 12:13:55 Waiting for app: TestExipryReplace to get deployed
2021/12/02 12:13:55 App: TestExipryReplace got deployed
2021/12/02 12:13:55 Starting bucket ops to source bucket
2021/12/02 12:14:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:14:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:14:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:14:40 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:14:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:14:45 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 12:14:50 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 12:14:55 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 12:15:00 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 12:15:00 App: TestExipryReplace got un-deployed
2021/12/02 12:15:00 No eventing-con process running
2021/12/02 12:15:00 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/02 12:15:02 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:15:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:15:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:15:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:15:14 Testing on the destination bucket for expiry insert
2021/12/02 12:15:14 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:15:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 12:15:20 Waiting for app: TestExipryReplace to get deployed
2021/12/02 12:15:25 Waiting for app: TestExipryReplace to get deployed
2021/12/02 12:15:30 Waiting for app: TestExipryReplace to get deployed
2021/12/02 12:15:35 Waiting for app: TestExipryReplace to get deployed
2021/12/02 12:15:35 App: TestExipryReplace got deployed
2021/12/02 12:15:35 Starting bucket ops to source bucket
2021/12/02 12:16:20 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:16:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:16:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:16:20 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:16:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:16:26 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 12:16:31 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 12:16:36 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 12:16:41 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 12:16:41 App: TestExipryReplace got un-deployed
2021/12/02 12:16:41 No eventing-con process running
2021/12/02 12:16:41 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/02 12:16:42 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:16:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:16:57 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:16:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:16:59 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:17:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.72s)
=== RUN   TestExipryUpsert
2021/12/02 12:17:04 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:17:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/02 12:17:10 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 12:17:15 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 12:17:20 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 12:17:25 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 12:17:25 App: TestExipryUpsert got deployed
2021/12/02 12:17:25 Starting bucket ops to source bucket
2021/12/02 12:18:11 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:18:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:18:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:18:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:18:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:18:16 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 12:18:21 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 12:18:26 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 12:18:31 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 12:18:31 App: TestExipryUpsert got un-deployed
2021/12/02 12:18:31 No eventing-con process running
2021/12/02 12:18:31 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/02 12:18:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:18:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:18:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:18:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:18:45 Testing on the destination bucket for expiry insert
2021/12/02 12:18:45 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:18:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/02 12:18:51 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 12:18:56 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 12:19:01 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 12:19:06 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 12:19:06 App: TestExipryUpsert got deployed
2021/12/02 12:19:06 Starting bucket ops to source bucket
2021/12/02 12:19:51 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:19:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:19:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:19:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:19:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:19:57 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 12:20:02 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 12:20:07 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 12:20:12 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 12:20:12 App: TestExipryUpsert got un-deployed
2021/12/02 12:20:12 No eventing-con process running
2021/12/02 12:20:12 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/02 12:20:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:20:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:20:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:20:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:20:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:20:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (210.97s)
=== RUN   TestCasReplace
2021/12/02 12:20:35 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:20:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 12:20:41 Waiting for app: TestCasReplace to get deployed
2021/12/02 12:20:46 Waiting for app: TestCasReplace to get deployed
2021/12/02 12:20:51 Waiting for app: TestCasReplace to get deployed
2021/12/02 12:20:56 Waiting for app: TestCasReplace to get deployed
2021/12/02 12:20:56 App: TestCasReplace got deployed
2021/12/02 12:20:56 Starting bucket ops to source bucket
2021/12/02 12:20:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:20:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:20:57 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:20:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:21:02 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 12:21:07 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 12:21:12 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 12:21:17 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 12:21:17 App: TestCasReplace got un-deployed
2021/12/02 12:21:17 No eventing-con process running
2021/12/02 12:21:17 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/02 12:21:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:21:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:21:26 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:21:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:21:31 Testing on the destination bucket for upsert with cas
2021/12/02 12:21:31 Starting bucket ops to source bucket
2021/12/02 12:21:31 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:21:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 12:21:37 Waiting for app: TestCasReplace to get deployed
2021/12/02 12:21:42 Waiting for app: TestCasReplace to get deployed
2021/12/02 12:21:47 Waiting for app: TestCasReplace to get deployed
2021/12/02 12:21:52 Waiting for app: TestCasReplace to get deployed
2021/12/02 12:21:52 App: TestCasReplace got deployed
2021/12/02 12:21:52 Starting bucket ops to source bucket
2021/12/02 12:21:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:22:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:22:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:22:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:22:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:22:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:22:07 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 12:22:12 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 12:22:17 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 12:22:22 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 12:22:22 App: TestCasReplace got un-deployed
2021/12/02 12:22:22 No eventing-con process running
2021/12/02 12:22:22 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/02 12:22:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:22:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:22:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:22:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:22:41 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:22:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.88s)
=== RUN   TestCasDelete
2021/12/02 12:22:46 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:22:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 12:22:52 Waiting for app: TestCasDelete to get deployed
2021/12/02 12:22:57 Waiting for app: TestCasDelete to get deployed
2021/12/02 12:23:02 Waiting for app: TestCasDelete to get deployed
2021/12/02 12:23:07 Waiting for app: TestCasDelete to get deployed
2021/12/02 12:23:07 App: TestCasDelete got deployed
2021/12/02 12:23:07 Starting bucket ops to source bucket
2021/12/02 12:23:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:23:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:23:07 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:23:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:23:13 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 12:23:18 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 12:23:23 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 12:23:28 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 12:23:28 App: TestCasDelete got un-deployed
2021/12/02 12:23:28 No eventing-con process running
2021/12/02 12:23:28 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/02 12:23:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:23:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:23:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:23:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:23:42 Testing on the destination bucket for delete with cas
2021/12/02 12:23:42 Starting bucket ops to source bucket
2021/12/02 12:23:42 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:23:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 12:23:48 Waiting for app: TestCasDelete to get deployed
2021/12/02 12:23:53 Waiting for app: TestCasDelete to get deployed
2021/12/02 12:23:58 Waiting for app: TestCasDelete to get deployed
2021/12/02 12:24:03 Waiting for app: TestCasDelete to get deployed
2021/12/02 12:24:03 App: TestCasDelete got deployed
2021/12/02 12:24:03 Starting bucket ops to source bucket
2021/12/02 12:24:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:24:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:24:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:24:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:24:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 12:24:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:24:18 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 12:24:23 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 12:24:28 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 12:24:33 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 12:24:33 App: TestCasDelete got un-deployed
2021/12/02 12:24:33 No eventing-con process running
2021/12/02 12:24:33 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/02 12:24:35 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:24:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:24:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:24:47 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:24:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (125.92s)
=== RUN   TestCountersIncrement
2021/12/02 12:24:52 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 12:24:58 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 12:24:58 Removing node(s): [] from the cluster
2021/12/02 12:25:03 Rebalance progress: 100
2021/12/02 12:25:03 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:25:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/02 12:25:09 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 12:25:14 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 12:25:19 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 12:25:24 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 12:25:24 App: TestCountersIncrement got deployed
2021/12/02 12:25:24 Starting bucket ops to source bucket
2021/12/02 12:25:30 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/02 12:25:36 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/02 12:25:36 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/02 12:25:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:25:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:25:41 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 12:25:46 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 12:25:51 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 12:25:56 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 12:25:56 App: TestCountersIncrement got un-deployed
2021/12/02 12:25:56 No eventing-con process running
2021/12/02 12:25:56 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/02 12:25:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:26:07 Waiting for dst bucket item count to get to: 0 curr count: 513
2021/12/02 12:26:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:26:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:26:19 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 12:26:24 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/02 12:26:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:26:24 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:26:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/02 12:26:31 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 12:26:36 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 12:26:41 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 12:26:46 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 12:26:46 App: TestCountersIncrement got deployed
2021/12/02 12:26:46 Starting bucket ops to source bucket
2021/12/02 12:26:52 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/02 12:26:52 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/02 12:26:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:26:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:26:57 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 12:27:02 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 12:27:07 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 12:27:12 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 12:27:12 App: TestCountersIncrement got un-deployed
2021/12/02 12:27:12 No eventing-con process running
2021/12/02 12:27:12 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/02 12:27:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:27:24 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/02 12:27:29 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 12:27:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:27:31 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:27:41 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/02 12:27:46 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/02 12:27:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:27:46 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/02 12:27:51 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.72s)
=== RUN   TestCountersDecrement
2021/12/02 12:27:51 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 12:27:51 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 12:27:51 Removing node(s): [] from the cluster
2021/12/02 12:27:56 Rebalance progress: 100
2021/12/02 12:27:56 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:27:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/02 12:28:02 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 12:28:07 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 12:28:13 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 12:28:18 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 12:28:18 App: TestCountersDecrement got deployed
2021/12/02 12:28:18 Starting bucket ops to source bucket
2021/12/02 12:28:24 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/02 12:28:29 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/02 12:28:34 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/02 12:28:34 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/02 12:28:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:28:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:28:39 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 12:28:44 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 12:28:49 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 12:28:54 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 12:28:54 App: TestCountersDecrement got un-deployed
2021/12/02 12:28:54 No eventing-con process running
2021/12/02 12:28:54 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/02 12:28:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:29:06 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/02 12:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:29:07 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:29:17 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/02 12:29:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:29:17 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:29:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/02 12:29:24 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 12:29:29 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 12:29:34 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 12:29:39 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 12:29:39 App: TestCountersDecrement got deployed
2021/12/02 12:29:39 Starting bucket ops to source bucket
2021/12/02 12:29:45 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/02 12:29:50 Waiting for dst bucket item count to get to: 2048 curr count: 429
2021/12/02 12:29:55 Waiting for dst bucket item count to get to: 2048 curr count: 1453
2021/12/02 12:29:55 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/02 12:29:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:29:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:30:00 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 12:30:05 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 12:30:10 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 12:30:15 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 12:30:15 App: TestCountersDecrement got un-deployed
2021/12/02 12:30:15 No eventing-con process running
2021/12/02 12:30:15 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/02 12:30:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:30:27 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/02 12:30:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:30:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:30:39 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/02 12:30:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:30:39 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/02 12:30:44 Rebalance progress: 100
--- PASS: TestCountersDecrement (173.16s)
=== RUN   TestCacheGet
2021/12/02 12:30:44 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:30:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/02 12:30:50 Waiting for app: TestCacheGet to get deployed
2021/12/02 12:30:55 Waiting for app: TestCacheGet to get deployed
2021/12/02 12:31:00 Waiting for app: TestCacheGet to get deployed
2021/12/02 12:31:05 Waiting for app: TestCacheGet to get deployed
2021/12/02 12:31:05 App: TestCacheGet got deployed
2021/12/02 12:31:05 Starting bucket ops to source bucket
2021/12/02 12:31:10 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/02 12:31:15 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/02 12:31:20 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/02 12:31:25 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/02 12:31:30 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/02 12:31:30 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/02 12:31:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:31:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:31:35 Waiting for app: TestCacheGet to get un-deployed
2021/12/02 12:31:40 Waiting for app: TestCacheGet to get un-deployed
2021/12/02 12:31:45 Waiting for app: TestCacheGet to get un-deployed
2021/12/02 12:31:50 Waiting for app: TestCacheGet to get un-deployed
2021/12/02 12:31:50 App: TestCacheGet got un-deployed
2021/12/02 12:31:50 No eventing-con process running
2021/12/02 12:31:50 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/02 12:31:52 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:32:02 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/02 12:32:07 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/02 12:32:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:32:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:32:19 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/02 12:32:24 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/02 12:32:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (100.19s)
=== RUN   TestCacheRYOW
2021/12/02 12:32:24 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:32:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/02 12:32:30 Waiting for app: TestCacheRYOW to get deployed
2021/12/02 12:32:35 Waiting for app: TestCacheRYOW to get deployed
2021/12/02 12:32:40 Waiting for app: TestCacheRYOW to get deployed
2021/12/02 12:32:45 Waiting for app: TestCacheRYOW to get deployed
2021/12/02 12:32:45 App: TestCacheRYOW got deployed
2021/12/02 12:32:45 Starting bucket ops to source bucket
2021/12/02 12:32:50 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/02 12:32:50 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/02 12:32:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:32:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:32:56 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/02 12:33:01 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/02 12:33:06 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/02 12:33:11 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/02 12:33:11 App: TestCacheRYOW got un-deployed
2021/12/02 12:33:11 No eventing-con process running
2021/12/02 12:33:11 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/02 12:33:12 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:33:22 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:33:27 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/02 12:33:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:33:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:33:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:33:44 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/02 12:33:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.28s)
=== RUN   TestCacheExpiry
2021/12/02 12:33:44 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:33:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/02 12:33:50 Waiting for app: TestCacheExpiry to get deployed
2021/12/02 12:33:56 Waiting for app: TestCacheExpiry to get deployed
2021/12/02 12:34:01 Waiting for app: TestCacheExpiry to get deployed
2021/12/02 12:34:06 Waiting for app: TestCacheExpiry to get deployed
2021/12/02 12:34:06 App: TestCacheExpiry got deployed
2021/12/02 12:34:06 Starting bucket ops to source bucket
2021/12/02 12:34:11 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 12:34:16 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/02 12:34:21 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/02 12:34:26 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/02 12:34:31 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/02 12:34:31 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 12:34:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:34:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:34:36 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/02 12:34:41 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/02 12:34:46 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/02 12:34:51 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/02 12:34:51 App: TestCacheExpiry got un-deployed
2021/12/02 12:34:51 No eventing-con process running
2021/12/02 12:34:51 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/02 12:34:53 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:35:03 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:35:08 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/02 12:35:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:35:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:35:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:35:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:35:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.25s)
=== RUN   TestCacheExpiryCustom
2021/12/02 12:35:25 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:35:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/02 12:35:31 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/02 12:35:36 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/02 12:35:41 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/02 12:35:46 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/02 12:35:46 App: TestCacheExpiryCustom got deployed
2021/12/02 12:35:46 Starting bucket ops to source bucket
2021/12/02 12:35:51 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 12:35:56 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/02 12:36:01 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/02 12:36:06 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/02 12:36:11 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/02 12:36:11 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 12:36:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:36:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:36:16 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/02 12:36:21 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/02 12:36:26 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/02 12:36:31 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/02 12:36:31 App: TestCacheExpiryCustom got un-deployed
2021/12/02 12:36:31 No eventing-con process running
2021/12/02 12:36:31 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/02 12:36:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:36:43 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:36:48 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/02 12:36:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:36:50 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:37:00 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/02 12:37:05 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/02 12:37:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.47s)
=== RUN   TestCacheOverflowCustom
2021/12/02 12:37:05 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:37:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/02 12:37:11 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/02 12:37:16 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/02 12:37:21 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/02 12:37:26 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/02 12:37:26 App: TestCacheOverflowCustom got deployed
2021/12/02 12:37:26 Starting bucket ops to source bucket
2021/12/02 12:37:31 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/02 12:37:36 Waiting for dst bucket item count to get to: 35 curr count: 10
2021/12/02 12:37:41 Waiting for dst bucket item count to get to: 35 curr count: 10
2021/12/02 12:37:46 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/02 12:37:51 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/02 12:37:51 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/02 12:37:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:37:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:37:56 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/02 12:38:01 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/02 12:38:06 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/02 12:38:11 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/02 12:38:12 App: TestCacheOverflowCustom got un-deployed
2021/12/02 12:38:12 No eventing-con process running
2021/12/02 12:38:12 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/02 12:38:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:38:23 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/02 12:38:28 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/02 12:38:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:38:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:38:40 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/02 12:38:45 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/02 12:38:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.30s)
=== RUN   TestStrictMode
2021/12/02 12:38:45 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:38:46 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/02 12:38:46 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:38:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/02 12:38:53 Waiting for app: TestDataTypes to get deployed
2021/12/02 12:38:58 Waiting for app: TestDataTypes to get deployed
2021/12/02 12:39:03 Waiting for app: TestDataTypes to get deployed
2021/12/02 12:39:08 Waiting for app: TestDataTypes to get deployed
2021/12/02 12:39:08 App: TestDataTypes got deployed
2021/12/02 12:39:08 Starting bucket ops to source bucket
2021/12/02 12:39:13 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/02 12:39:18 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/02 12:39:23 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/02 12:39:23 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/02 12:39:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:39:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:39:28 Waiting for app: TestDataTypes to get un-deployed
2021/12/02 12:39:33 Waiting for app: TestDataTypes to get un-deployed
2021/12/02 12:39:38 Waiting for app: TestDataTypes to get un-deployed
2021/12/02 12:39:43 Waiting for app: TestDataTypes to get un-deployed
2021/12/02 12:39:43 App: TestDataTypes got un-deployed
2021/12/02 12:39:43 No eventing-con process running
2021/12/02 12:39:43 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/02 12:39:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:39:55 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/02 12:39:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:39:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:40:06 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/02 12:40:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.10s)
=== RUN   TestEnoentGet
2021/12/02 12:40:06 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:40:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/02 12:40:13 Waiting for app: TestEnoentGet to get deployed
2021/12/02 12:40:18 Waiting for app: TestEnoentGet to get deployed
2021/12/02 12:40:23 Waiting for app: TestEnoentGet to get deployed
2021/12/02 12:40:28 Waiting for app: TestEnoentGet to get deployed
2021/12/02 12:40:28 App: TestEnoentGet got deployed
2021/12/02 12:40:28 Starting bucket ops to source bucket
2021/12/02 12:40:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 12:40:38 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 12:40:43 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 12:40:43 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 12:40:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:40:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:40:48 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 12:40:53 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 12:40:58 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 12:41:03 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 12:41:03 App: TestEnoentGet got un-deployed
2021/12/02 12:41:03 No eventing-con process running
2021/12/02 12:41:03 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/02 12:41:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:41:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:41:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:41:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:41:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:41:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:41:27 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:41:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/02 12:41:33 Waiting for app: TestEnoentGet to get deployed
2021/12/02 12:41:38 Waiting for app: TestEnoentGet to get deployed
2021/12/02 12:41:43 Waiting for app: TestEnoentGet to get deployed
2021/12/02 12:41:48 Waiting for app: TestEnoentGet to get deployed
2021/12/02 12:41:48 App: TestEnoentGet got deployed
2021/12/02 12:41:48 Starting bucket ops to source bucket
2021/12/02 12:41:53 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 12:41:58 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 12:42:03 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 12:42:03 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 12:42:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:42:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:42:08 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 12:42:13 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 12:42:18 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 12:42:23 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 12:42:23 App: TestEnoentGet got un-deployed
2021/12/02 12:42:23 No eventing-con process running
2021/12/02 12:42:23 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/02 12:42:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:42:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:42:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:42:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:42:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:42:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.40s)
=== RUN   TestEnoentDelete
2021/12/02 12:42:47 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:42:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 12:42:53 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 12:42:58 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 12:43:03 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 12:43:08 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 12:43:08 App: TestEnoentDelete got deployed
2021/12/02 12:43:08 Starting bucket ops to source bucket
2021/12/02 12:43:13 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 12:43:18 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 12:43:18 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 12:43:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:43:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:43:23 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 12:43:28 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 12:43:33 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 12:43:38 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 12:43:38 App: TestEnoentDelete got un-deployed
2021/12/02 12:43:38 No eventing-con process running
2021/12/02 12:43:38 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/02 12:43:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:43:50 Waiting for dst bucket item count to get to: 0 curr count: 71
2021/12/02 12:43:55 Waiting for dst bucket item count to get to: 0 curr count: 71
2021/12/02 12:43:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:43:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:44:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:44:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:44:07 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:44:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 12:44:13 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 12:44:18 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 12:44:23 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 12:44:28 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 12:44:28 App: TestEnoentDelete got deployed
2021/12/02 12:44:28 Starting bucket ops to source bucket
2021/12/02 12:44:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 12:44:38 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 12:44:38 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 12:44:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:44:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:44:44 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 12:44:49 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 12:44:54 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 12:44:59 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 12:44:59 App: TestEnoentDelete got un-deployed
2021/12/02 12:44:59 No eventing-con process running
2021/12/02 12:44:59 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/02 12:45:00 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:45:10 Waiting for dst bucket item count to get to: 0 curr count: 78
2021/12/02 12:45:15 Waiting for dst bucket item count to get to: 0 curr count: 78
2021/12/02 12:45:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:45:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:45:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:45:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.41s)
=== RUN   TestError
2021/12/02 12:45:37 Waiting for app: TestError to get un-deployed
2021/12/02 12:45:37 App: TestError got un-deployed
2021/12/02 12:45:37 No eventing-con process running
2021/12/02 12:45:37 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/02 12:45:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:45:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:45:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:45:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:45:51 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:45:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/02 12:45:57 Waiting for app: TestError to get deployed
2021/12/02 12:46:02 Waiting for app: TestError to get deployed
2021/12/02 12:46:07 Waiting for app: TestError to get deployed
2021/12/02 12:46:12 Waiting for app: TestError to get deployed
2021/12/02 12:46:12 App: TestError got deployed
2021/12/02 12:46:12 Starting bucket ops to source bucket
2021/12/02 12:46:17 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/02 12:46:22 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/02 12:46:22 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/02 12:46:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:46:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:46:28 Waiting for app: TestError to get un-deployed
2021/12/02 12:46:33 Waiting for app: TestError to get un-deployed
2021/12/02 12:46:38 Waiting for app: TestError to get un-deployed
2021/12/02 12:46:43 Waiting for app: TestError to get un-deployed
2021/12/02 12:46:43 App: TestError got un-deployed
2021/12/02 12:46:43 No eventing-con process running
2021/12/02 12:46:43 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/02 12:46:44 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:46:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:46:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:46:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:47:06 Waiting for dst bucket item count to get to: 0 curr count: 300
2021/12/02 12:47:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (98.97s)
=== RUN   TestCRLF
2021/12/02 12:47:16 Waiting for app: TestCRLF to get un-deployed
2021/12/02 12:47:16 App: TestCRLF got un-deployed
2021/12/02 12:47:16 No eventing-con process running
2021/12/02 12:47:16 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/02 12:47:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:47:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:47:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:47:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:47:30 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:47:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/02 12:47:36 Waiting for app: TestCRLF to get deployed
2021/12/02 12:47:41 Waiting for app: TestCRLF to get deployed
2021/12/02 12:47:46 Waiting for app: TestCRLF to get deployed
2021/12/02 12:47:51 Waiting for app: TestCRLF to get deployed
2021/12/02 12:47:51 App: TestCRLF got deployed
2021/12/02 12:47:51 Starting bucket ops to source bucket
2021/12/02 12:47:56 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 12:48:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 12:48:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 12:48:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:48:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:48:07 Waiting for app: TestCRLF to get un-deployed
2021/12/02 12:48:12 Waiting for app: TestCRLF to get un-deployed
2021/12/02 12:48:17 Waiting for app: TestCRLF to get un-deployed
2021/12/02 12:48:22 Waiting for app: TestCRLF to get un-deployed
2021/12/02 12:48:22 App: TestCRLF got un-deployed
2021/12/02 12:48:22 No eventing-con process running
2021/12/02 12:48:22 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/02 12:48:23 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:48:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 12:48:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 12:48:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:48:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:48:50 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/02 12:48:55 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/02 12:48:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.97s)
=== RUN   TestImportExport
2021/12/02 12:49:05 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 12:49:05 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/02 12:49:05 No eventing-con process running
2021/12/02 12:49:05 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/02 12:49:07 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:49:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:49:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:49:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:49:24 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 12:49:24 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/02 12:49:24 No eventing-con process running
2021/12/02 12:49:24 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/02 12:49:26 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:49:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:49:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:49:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:49:38 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:49:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 12:49:39 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:49:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 12:49:45 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/02 12:49:50 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/02 12:49:55 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/02 12:50:00 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/02 12:50:00 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/02 12:50:05 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/02 12:50:10 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/02 12:50:15 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/02 12:50:15 App: TestImportExport_bucket_op_on_update got deployed
2021/12/02 12:50:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:50:20 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 12:50:25 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 12:50:30 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 12:50:35 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 12:50:35 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/02 12:50:35 No eventing-con process running
2021/12/02 12:50:35 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/02 12:50:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:50:40 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 12:50:45 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 12:50:51 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 12:50:56 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 12:50:56 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/02 12:50:56 No eventing-con process running
2021/12/02 12:50:56 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/02 12:51:13 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 12:51:13 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/02 12:51:13 No eventing-con process running
2021/12/02 12:51:13 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/02 12:51:15 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:51:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:51:22 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:51:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:51:32 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 12:51:32 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/02 12:51:32 No eventing-con process running
2021/12/02 12:51:32 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/02 12:51:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:51:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:51:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:51:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (170.07s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/02 12:51:55 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:51:55 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 12:51:55 No eventing-con process running
2021/12/02 12:51:55 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/02 12:51:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:52:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:52:04 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:52:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:52:09 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:52:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 12:52:10 Starting bucket ops to source bucket
2021/12/02 12:52:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:52:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:52:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:52:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:52:38 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/02 12:52:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 12:52:43 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/02 12:52:43 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/02 12:52:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:52:43 Undeploying app: bucket_op_on_update
2021/12/02 12:52:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:52:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:52:53 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:52:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:53:03 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:53:03 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 12:53:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:53:03 No eventing-con process running
2021/12/02 12:53:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:53:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:53:48 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:53:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 12:53:49 Starting bucket ops to source bucket
2021/12/02 12:53:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:54:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:54:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:54:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:54:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/02 12:54:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 12:54:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/02 12:54:22 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/02 12:54:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:54:22 Undeploying app: bucket_op_on_update
2021/12/02 12:54:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:54:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:54:32 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:54:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:54:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:54:42 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 12:54:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:54:42 No eventing-con process running
2021/12/02 12:54:44 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:54:51 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:55:26 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:55:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 12:55:27 Starting bucket ops to source bucket
2021/12/02 12:55:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:55:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:55:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:55:50 Waiting for dst bucket item count to get to: 5000 curr count: 108
2021/12/02 12:55:55 Waiting for dst bucket item count to get to: 5000 curr count: 108
2021/12/02 12:55:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 12:56:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/02 12:56:00 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/02 12:56:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:56:00 Undeploying app: bucket_op_on_update
2021/12/02 12:56:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:56:05 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:56:11 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:56:16 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:56:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:56:21 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 12:56:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:56:21 No eventing-con process running
2021/12/02 12:56:22 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:56:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:57:05 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:57:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 12:57:06 Starting bucket ops to source bucket
2021/12/02 12:57:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:57:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:57:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:57:29 Waiting for dst bucket item count to get to: 5000 curr count: 2393
2021/12/02 12:57:34 Waiting for dst bucket item count to get to: 5000 curr count: 2393
2021/12/02 12:57:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 12:57:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/02 12:57:39 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/02 12:57:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:57:39 Undeploying app: bucket_op_on_update
2021/12/02 12:57:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:57:44 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:57:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:57:54 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:57:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:57:59 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 12:57:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:57:59 No eventing-con process running
2021/12/02 12:58:01 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:58:08 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:58:43 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 12:58:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 12:58:44 Starting bucket ops to source bucket
2021/12/02 12:58:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:58:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:59:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 12:59:07 Waiting for dst bucket item count to get to: 5000 curr count: 1369
2021/12/02 12:59:12 Waiting for dst bucket item count to get to: 5000 curr count: 3869
2021/12/02 12:59:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 12:59:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/02 12:59:17 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/02 12:59:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 12:59:18 Undeploying app: bucket_op_on_update
2021/12/02 12:59:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 12:59:23 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:59:28 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:59:33 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:59:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 12:59:38 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 12:59:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 12:59:38 No eventing-con process running
2021/12/02 12:59:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 12:59:47 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:00:22 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (516.58s)
=== RUN   TestOnUpdateN1QLOp
2021/12/02 13:00:32 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/02 13:00:32 App: TestOnUpdateN1QLOp got un-deployed
2021/12/02 13:00:32 No eventing-con process running
2021/12/02 13:00:32 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/02 13:00:34 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:00:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:00:41 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:00:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:00:46 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:00:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/02 13:00:47 Starting bucket ops to source bucket
2021/12/02 13:00:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:01:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:01:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:01:10 Waiting for dst bucket item count to get to: 5000 curr count: 263
2021/12/02 13:01:15 Waiting for dst bucket item count to get to: 5000 curr count: 263
2021/12/02 13:01:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:01:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:01:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:01:20 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/02 13:01:25 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/02 13:01:30 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/02 13:01:35 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/02 13:01:35 App: TestOnUpdateN1QLOp got un-deployed
2021/12/02 13:01:35 No eventing-con process running
2021/12/02 13:01:35 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/02 13:01:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:01:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:01:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:01:49 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:01:59 Waiting for dst bucket item count to get to: 0 curr count: 4527
2021/12/02 13:02:04 Waiting for dst bucket item count to get to: 0 curr count: 2033
2021/12/02 13:02:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.82s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/02 13:02:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/02 13:02:14 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/02 13:02:14 No eventing-con process running
2021/12/02 13:02:14 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/02 13:02:15 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:02:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:02:22 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:02:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:02:27 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:02:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 13:02:28 Starting bucket ops to source bucket
2021/12/02 13:02:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:02:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:02:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:02:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:02:56 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/02 13:02:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:02:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:02:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:03:01 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/02 13:03:06 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/02 13:03:11 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/02 13:03:17 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/02 13:03:17 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/02 13:03:17 No eventing-con process running
2021/12/02 13:03:17 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/02 13:03:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:03:28 Waiting for dst bucket item count to get to: 0 curr count: 3979
2021/12/02 13:03:33 Waiting for dst bucket item count to get to: 0 curr count: 1479
2021/12/02 13:03:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:03:35 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:03:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:03:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.53s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/02 13:03:55 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 13:03:55 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/02 13:03:55 No eventing-con process running
2021/12/02 13:03:55 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/02 13:03:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:04:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:04:04 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:04:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:04:09 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:04:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 13:04:10 Starting bucket ops to source bucket
2021/12/02 13:04:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:04:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:04:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:04:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:04:38 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/02 13:04:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:04:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:04:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:04:43 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 13:04:48 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 13:04:53 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 13:04:58 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 13:04:58 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/02 13:04:58 No eventing-con process running
2021/12/02 13:04:58 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/02 13:05:00 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:05:10 Waiting for dst bucket item count to get to: 0 curr count: 3078
2021/12/02 13:05:15 Waiting for dst bucket item count to get to: 0 curr count: 3078
2021/12/02 13:05:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:05:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:05:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:05:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.73s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/02 13:05:37 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/02 13:05:37 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/02 13:05:37 No eventing-con process running
2021/12/02 13:05:37 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/02 13:05:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:05:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:05:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:05:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:05:51 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:05:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/02 13:05:52 Starting bucket ops to source bucket
2021/12/02 13:06:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:06:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:06:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:06:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:06:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:06:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:06:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:06:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/02 13:06:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/02 13:06:30 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/02 13:06:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/02 13:06:35 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/02 13:06:35 No eventing-con process running
2021/12/02 13:06:35 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/02 13:06:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:06:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:06:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:06:49 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:06:59 Waiting for dst bucket item count to get to: 0 curr count: 4492
2021/12/02 13:07:04 Waiting for dst bucket item count to get to: 0 curr count: 1992
2021/12/02 13:07:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (96.76s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/02 13:07:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/02 13:07:14 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/02 13:07:14 No eventing-con process running
2021/12/02 13:07:14 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/02 13:07:16 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:07:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:07:22 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:07:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:07:27 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:07:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/02 13:07:29 Starting bucket ops to source bucket
2021/12/02 13:07:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:07:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:07:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:07:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:07:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/02 13:07:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:07:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:07:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:08:02 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/02 13:08:07 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/02 13:08:12 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/02 13:08:17 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/02 13:08:17 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/02 13:08:17 No eventing-con process running
2021/12/02 13:08:17 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/02 13:08:19 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:08:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:08:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 13:08:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:08:36 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:08:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:08:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.28s)
=== RUN   TestOnDeleteBucketOp
2021/12/02 13:08:56 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/02 13:08:56 App: TestOnDeleteBucketOp got un-deployed
2021/12/02 13:08:56 No eventing-con process running
2021/12/02 13:08:56 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/02 13:08:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:09:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:09:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:09:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:09:10 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:09:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 13:09:11 Starting bucket ops to source bucket
2021/12/02 13:09:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:09:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:09:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:09:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:09:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:09:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:09:37 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/02 13:09:42 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/02 13:09:47 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/02 13:09:52 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/02 13:09:52 App: TestOnDeleteBucketOp got un-deployed
2021/12/02 13:09:52 No eventing-con process running
2021/12/02 13:09:52 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/02 13:09:54 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:09:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:10:01 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:10:11 Waiting for dst bucket item count to get to: 0 curr count: 4469
2021/12/02 13:10:16 Waiting for dst bucket item count to get to: 0 curr count: 4469
2021/12/02 13:10:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (90.32s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/02 13:10:26 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/02 13:10:26 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/02 13:10:26 No eventing-con process running
2021/12/02 13:10:26 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/02 13:10:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:10:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:10:35 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:10:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:10:40 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:10:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/02 13:10:41 Starting bucket ops to source bucket
2021/12/02 13:10:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:10:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:11:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:11:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:11:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:11:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:11:07 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/02 13:11:12 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/02 13:11:17 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/02 13:11:22 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/02 13:11:22 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/02 13:11:22 No eventing-con process running
2021/12/02 13:11:22 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/02 13:11:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:11:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:11:31 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:11:41 Waiting for dst bucket item count to get to: 0 curr count: 4853
2021/12/02 13:11:47 Waiting for dst bucket item count to get to: 0 curr count: 4853
2021/12/02 13:11:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.22s)
=== RUN   TestTimerBucketOp
2021/12/02 13:11:57 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/02 13:11:57 App: TestTimerBucketOp got un-deployed
2021/12/02 13:11:57 No eventing-con process running
2021/12/02 13:11:57 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/02 13:11:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:12:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:12:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:12:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:12:10 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:12:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 13:12:11 Starting bucket ops to source bucket
2021/12/02 13:12:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:12:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:12:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:12:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:12:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:12:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:12:49 Waiting for dst bucket item count to get to: 5000 curr count: 1731
2021/12/02 13:12:54 Waiting for dst bucket item count to get to: 5000 curr count: 1731
2021/12/02 13:12:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:12:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:12:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:12:59 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/02 13:13:04 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/02 13:13:09 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/02 13:13:14 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/02 13:13:14 App: TestTimerBucketOp got un-deployed
2021/12/02 13:13:14 No eventing-con process running
2021/12/02 13:13:14 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/02 13:13:16 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:13:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:13:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:13:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:13:38 Waiting for dst bucket item count to get to: 0 curr count: 3792
2021/12/02 13:13:43 Waiting for dst bucket item count to get to: 0 curr count: 1292
2021/12/02 13:13:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.60s)
=== RUN   TestTimerInPastBucketOp
2021/12/02 13:13:53 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/02 13:13:53 App: TestTimerInPastBucketOp got un-deployed
2021/12/02 13:13:53 No eventing-con process running
2021/12/02 13:13:53 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/02 13:13:55 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:14:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:14:02 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:14:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:14:07 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:14:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 13:14:08 Starting bucket ops to source bucket
2021/12/02 13:14:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:14:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:14:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:14:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:14:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:14:41 Waiting for dst bucket item count to get to: 5000 curr count: 544
2021/12/02 13:14:46 Waiting for dst bucket item count to get to: 5000 curr count: 544
2021/12/02 13:14:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:14:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:14:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:14:51 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/02 13:14:56 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/02 13:15:01 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/02 13:15:06 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/02 13:15:06 App: TestTimerInPastBucketOp got un-deployed
2021/12/02 13:15:06 No eventing-con process running
2021/12/02 13:15:06 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/02 13:15:08 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:15:18 Waiting for dst bucket item count to get to: 0 curr count: 3356
2021/12/02 13:15:23 Waiting for dst bucket item count to get to: 0 curr count: 856
2021/12/02 13:15:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:15:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:15:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:15:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.81s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/02 13:15:45 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:15:45 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 13:15:45 No eventing-con process running
2021/12/02 13:15:45 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/02 13:15:47 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:15:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:15:54 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:15:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:15:59 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:16:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 13:16:00 Starting bucket ops to source bucket
2021/12/02 13:16:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:16:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:16:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:16:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:16:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:16:33 Waiting for dst bucket item count to get to: 5000 curr count: 287
2021/12/02 13:16:38 Waiting for dst bucket item count to get to: 5000 curr count: 2072
2021/12/02 13:16:43 Waiting for dst bucket item count to get to: 5000 curr count: 4285
2021/12/02 13:16:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:16:48 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/02 13:16:48 App: TestDeployUndeployLoopTimer got deployed
2021/12/02 13:16:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:16:48 Undeploying app: bucket_op_with_timer
2021/12/02 13:16:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:16:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:16:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:17:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:17:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:17:08 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 13:17:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:17:08 No eventing-con process running
2021/12/02 13:17:10 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:17:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:17:52 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:17:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 13:17:53 Starting bucket ops to source bucket
2021/12/02 13:18:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:18:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:18:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:18:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:18:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:18:26 Waiting for dst bucket item count to get to: 5000 curr count: 16
2021/12/02 13:18:31 Waiting for dst bucket item count to get to: 5000 curr count: 20
2021/12/02 13:18:36 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/02 13:18:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:18:41 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/02 13:18:41 App: TestDeployUndeployLoopTimer got deployed
2021/12/02 13:18:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:18:41 Undeploying app: bucket_op_with_timer
2021/12/02 13:18:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:18:46 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:18:51 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:18:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:19:01 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:19:01 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 13:19:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:19:01 No eventing-con process running
2021/12/02 13:19:03 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:19:11 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:19:46 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:19:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 13:19:47 Starting bucket ops to source bucket
2021/12/02 13:19:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:20:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:20:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:20:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:20:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:20:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:20:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:20:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:20:30 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/02 13:20:30 App: TestDeployUndeployLoopTimer got deployed
2021/12/02 13:20:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:20:30 Undeploying app: bucket_op_with_timer
2021/12/02 13:20:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:20:35 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:20:40 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:20:45 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:20:50 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:20:50 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 13:20:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:20:50 No eventing-con process running
2021/12/02 13:20:52 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:20:59 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:21:34 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:21:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 13:21:35 Starting bucket ops to source bucket
2021/12/02 13:21:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:21:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:21:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:21:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:22:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:22:08 Waiting for dst bucket item count to get to: 5000 curr count: 1413
2021/12/02 13:22:13 Waiting for dst bucket item count to get to: 5000 curr count: 1413
2021/12/02 13:22:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:22:18 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/02 13:22:18 App: TestDeployUndeployLoopTimer got deployed
2021/12/02 13:22:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:22:18 Undeploying app: bucket_op_with_timer
2021/12/02 13:22:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:22:23 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:22:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:22:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:22:38 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:22:38 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 13:22:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:22:39 No eventing-con process running
2021/12/02 13:22:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:22:48 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:23:23 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:23:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 13:23:24 Starting bucket ops to source bucket
2021/12/02 13:23:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:23:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:23:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:23:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:23:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:23:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:24:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:24:07 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/02 13:24:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:24:12 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/02 13:24:12 App: TestDeployUndeployLoopTimer got deployed
2021/12/02 13:24:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:24:12 Undeploying app: bucket_op_with_timer
2021/12/02 13:24:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:24:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:24:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:24:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:24:32 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 13:24:32 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 13:24:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:24:32 No eventing-con process running
2021/12/02 13:24:34 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:24:41 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:25:16 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (581.26s)
=== RUN   TestMultipleHandlers
2021/12/02 13:25:26 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/02 13:25:26 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/02 13:25:26 No eventing-con process running
2021/12/02 13:25:26 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/02 13:25:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:25:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:25:35 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:25:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:25:45 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/02 13:25:45 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/02 13:25:45 No eventing-con process running
2021/12/02 13:25:45 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/02 13:25:47 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:25:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:25:53 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:25:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:25:59 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:26:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 13:26:00 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:26:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/02 13:26:06 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/02 13:26:11 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/02 13:26:16 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/02 13:26:21 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/02 13:26:21 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/02 13:26:26 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/02 13:26:31 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/02 13:26:36 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/02 13:26:36 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/02 13:26:36 Starting bucket ops to source bucket
2021/12/02 13:26:46 Waiting for dst bucket item count to get to: 5000 curr count: 648
2021/12/02 13:26:51 Waiting for dst bucket item count to get to: 5000 curr count: 2367
2021/12/02 13:26:56 Waiting for dst bucket item count to get to: 5000 curr count: 4219
2021/12/02 13:26:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:26:56 Starting bucket ops to source bucket
2021/12/02 13:27:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:27:11 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 13:27:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:27:11 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:27:11 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:27:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:27:16 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/02 13:27:21 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/02 13:27:26 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/02 13:27:31 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/02 13:27:31 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/02 13:27:31 No eventing-con process running
2021/12/02 13:27:31 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/02 13:27:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:27:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:27:41 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:27:51 Waiting for dst bucket item count to get to: 0 curr count: 7262
2021/12/02 13:27:56 Waiting for dst bucket item count to get to: 0 curr count: 7262
2021/12/02 13:27:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:27:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:28:01 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/02 13:28:06 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/02 13:28:11 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/02 13:28:16 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/02 13:28:16 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/02 13:28:16 No eventing-con process running
2021/12/02 13:28:16 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/02 13:28:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:28:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:28:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:28:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (193.29s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/02 13:28:39 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/02 13:28:39 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/02 13:28:40 No eventing-con process running
2021/12/02 13:28:40 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/02 13:28:41 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:28:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:28:48 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:28:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:28:53 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:28:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 13:28:54 Starting bucket ops to source bucket
2021/12/02 13:29:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:29:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:29:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:29:17 Waiting for dst bucket item count to get to: 5000 curr count: 1631
2021/12/02 13:29:22 Waiting for dst bucket item count to get to: 5000 curr count: 4131
2021/12/02 13:29:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:29:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:29:27 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/02 13:29:27 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/02 13:29:27 Pausing app: bucket_op_on_update
2021/12/02 13:29:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:29:32 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/02 13:29:32 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/02 13:29:32 Resuming app: bucket_op_on_update
2021/12/02 13:29:34 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:29:34 Starting bucket ops to source bucket
2021/12/02 13:29:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:29:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:29:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:29:57 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/02 13:29:57 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 13:29:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:30:02 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/02 13:30:02 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/02 13:30:02 Pausing app: bucket_op_on_update
2021/12/02 13:30:02 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:30:07 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/02 13:30:07 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/02 13:30:07 Resuming app: bucket_op_on_update
2021/12/02 13:30:08 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:30:08 Starting bucket ops to source bucket
2021/12/02 13:30:16 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 13:30:21 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 13:30:21 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/02 13:30:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:30:26 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/02 13:30:26 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/02 13:30:26 Pausing app: bucket_op_on_update
2021/12/02 13:30:26 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:30:31 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/02 13:30:31 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/02 13:30:31 Resuming app: bucket_op_on_update
2021/12/02 13:30:32 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:30:32 Starting bucket ops to source bucket
2021/12/02 13:30:40 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 13:30:45 Waiting for dst bucket item count to get to: 20000 curr count: 15567
2021/12/02 13:30:50 Waiting for dst bucket item count to get to: 20000 curr count: 15567
2021/12/02 13:30:55 Waiting for dst bucket item count to get to: 20000 curr count: 15567
2021/12/02 13:30:55 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/02 13:30:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:31:00 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/02 13:31:00 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/02 13:31:00 Pausing app: bucket_op_on_update
2021/12/02 13:31:00 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:31:06 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/02 13:31:06 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/02 13:31:06 Resuming app: bucket_op_on_update
2021/12/02 13:31:07 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:31:07 Starting bucket ops to source bucket
2021/12/02 13:31:15 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 13:31:20 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 13:31:25 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 13:31:25 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/02 13:31:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:31:30 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/02 13:31:30 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/02 13:31:30 Pausing app: bucket_op_on_update
2021/12/02 13:31:30 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:31:35 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/02 13:31:35 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/02 13:31:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:31:40 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/02 13:31:45 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/02 13:31:50 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/02 13:31:55 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/02 13:31:55 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/02 13:31:55 No eventing-con process running
2021/12/02 13:31:55 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/02 13:31:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:32:07 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/02 13:32:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:32:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:32:19 Waiting for dst bucket item count to get to: 0 curr count: 21890
2021/12/02 13:32:24 Waiting for dst bucket item count to get to: 0 curr count: 9390
2021/12/02 13:32:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (234.12s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/02 13:32:34 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/02 13:32:34 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/02 13:32:34 No eventing-con process running
2021/12/02 13:32:34 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/02 13:32:35 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:32:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:32:42 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:32:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:32:47 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:32:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 13:32:48 Starting bucket ops to source bucket
2021/12/02 13:32:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:33:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:33:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:33:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:33:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:33:21 Waiting for dst bucket item count to get to: 5000 curr count: 204
2021/12/02 13:33:26 Waiting for dst bucket item count to get to: 5000 curr count: 2704
2021/12/02 13:33:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:33:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:33:31 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/02 13:33:31 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/02 13:33:31 Pausing app: bucket_op_with_timer
2021/12/02 13:33:32 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:33:37 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/02 13:33:37 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/02 13:33:37 Resuming app: bucket_op_with_timer
2021/12/02 13:33:38 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:33:38 Starting bucket ops to source bucket
2021/12/02 13:33:46 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:33:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:33:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:34:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:34:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:34:11 Waiting for dst bucket item count to get to: 10000 curr count: 5356
2021/12/02 13:34:16 Waiting for dst bucket item count to get to: 10000 curr count: 7856
2021/12/02 13:34:16 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 13:34:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:34:21 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/02 13:34:21 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/02 13:34:21 Pausing app: bucket_op_with_timer
2021/12/02 13:34:21 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:34:26 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/02 13:34:26 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/02 13:34:26 Resuming app: bucket_op_with_timer
2021/12/02 13:34:27 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:34:27 Starting bucket ops to source bucket
2021/12/02 13:34:35 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 13:34:40 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 13:34:45 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 13:34:50 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 13:34:55 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 13:35:00 Waiting for dst bucket item count to get to: 15000 curr count: 11172
2021/12/02 13:35:00 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/02 13:35:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:35:05 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/02 13:35:05 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/02 13:35:05 Pausing app: bucket_op_with_timer
2021/12/02 13:35:06 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:35:11 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/02 13:35:11 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/02 13:35:11 Resuming app: bucket_op_with_timer
2021/12/02 13:35:12 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:35:12 Starting bucket ops to source bucket
2021/12/02 13:35:20 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 13:35:25 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 13:35:30 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 13:35:35 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 13:35:40 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 13:35:45 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 13:35:50 Waiting for dst bucket item count to get to: 20000 curr count: 19676
2021/12/02 13:35:55 Waiting for dst bucket item count to get to: 20000 curr count: 19676
2021/12/02 13:35:55 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/02 13:35:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:36:00 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/02 13:36:00 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/02 13:36:00 Pausing app: bucket_op_with_timer
2021/12/02 13:36:00 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:36:05 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/02 13:36:05 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/02 13:36:05 Resuming app: bucket_op_with_timer
2021/12/02 13:36:06 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:36:06 Starting bucket ops to source bucket
2021/12/02 13:36:14 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 13:36:19 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 13:36:24 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 13:36:29 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 13:36:34 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 13:36:39 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/02 13:36:44 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/02 13:36:44 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/02 13:36:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:36:49 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/02 13:36:49 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/02 13:36:49 Pausing app: bucket_op_with_timer
2021/12/02 13:36:49 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:36:54 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/02 13:36:54 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/02 13:36:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:36:59 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/02 13:37:04 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/02 13:37:09 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/02 13:37:14 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/02 13:37:14 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/02 13:37:14 No eventing-con process running
2021/12/02 13:37:14 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/02 13:37:16 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:37:26 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/02 13:37:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:37:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:37:38 Waiting for dst bucket item count to get to: 0 curr count: 17776
2021/12/02 13:37:43 Waiting for dst bucket item count to get to: 0 curr count: 5276
2021/12/02 13:37:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.44s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/02 13:37:53 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/02 13:37:53 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/02 13:37:53 No eventing-con process running
2021/12/02 13:37:53 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/02 13:37:55 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:38:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:38:02 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:38:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:38:07 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:38:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/02 13:38:13 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/02 13:38:18 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/02 13:38:23 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/02 13:38:28 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/02 13:38:28 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/02 13:38:28 Starting bucket ops to source bucket
2021/12/02 13:38:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:38:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:38:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:38:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:38:57 Waiting for dst bucket item count to get to: 5000 curr count: 843
2021/12/02 13:39:02 Waiting for dst bucket item count to get to: 5000 curr count: 2730
2021/12/02 13:39:07 Waiting for dst bucket item count to get to: 5000 curr count: 4387
2021/12/02 13:39:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:39:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:39:07 Pausing app: bucket_op_with_timer
2021/12/02 13:39:08 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:39:13 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/02 13:39:13 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/02 13:39:13 Resuming app: bucket_op_with_timer
2021/12/02 13:39:14 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:39:14 Starting bucket ops to source bucket
2021/12/02 13:39:26 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:39:31 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:39:36 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:39:41 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:39:46 Waiting for dst bucket item count to get to: 10000 curr count: 6302
2021/12/02 13:39:51 Waiting for dst bucket item count to get to: 10000 curr count: 8802
2021/12/02 13:39:51 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 13:39:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:39:56 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/02 13:40:01 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/02 13:40:06 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/02 13:40:11 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/02 13:40:11 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/02 13:40:11 No eventing-con process running
2021/12/02 13:40:11 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/02 13:40:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:40:23 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 13:40:28 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:40:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:40:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:40:40 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:40:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:40:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (181.61s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/02 13:40:55 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/02 13:40:55 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/02 13:40:55 No eventing-con process running
2021/12/02 13:40:55 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/02 13:40:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:41:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:41:03 Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:41:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:41:08 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:41:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/02 13:41:14 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/02 13:41:19 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/02 13:41:24 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/02 13:41:29 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/02 13:41:29 App: TestPauseResumeWithEventingReb got deployed
2021/12/02 13:41:29 Starting bucket ops to source bucket
2021/12/02 13:41:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:41:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:41:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:41:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:41:59 Waiting for dst bucket item count to get to: 5000 curr count: 1175
2021/12/02 13:42:04 Waiting for dst bucket item count to get to: 5000 curr count: 2349
2021/12/02 13:42:09 Waiting for dst bucket item count to get to: 5000 curr count: 3674
2021/12/02 13:42:09 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:42:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:42:09 Pausing app: bucket_op_with_timer
2021/12/02 13:42:09 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:42:14 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/02 13:42:14 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/02 13:42:14 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 13:42:14 Removing node(s): [] from the cluster
2021/12/02 13:42:19 Rebalance progress: 29.16666666666666
2021/12/02 13:42:24 Rebalance progress: 29.16666666666666
2021/12/02 13:42:29 Rebalance progress: 100
2021/12/02 13:42:29 Resuming app: bucket_op_with_timer
2021/12/02 13:42:30 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:42:30 Starting bucket ops to source bucket
2021/12/02 13:42:43 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:42:48 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:42:53 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:42:58 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:43:03 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 13:43:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 13:43:03 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 13:43:09 Rebalance progress: 29.16666666666666
2021/12/02 13:43:14 Rebalance progress: 29.16666666666666
2021/12/02 13:43:19 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/02 13:43:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 13:43:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 13:43:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 13:44:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 13:44:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 13:44:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 13:44:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 13:45:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 13:45:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 13:45:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 13:45:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: cfa5e75e0e20c14c188e16f4a8d61c49
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
Producer node: 172.31.2.134:9300	Node UUID: 19888b2dca7ad8d8a51006896a74538b
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
Producer node: 127.0.0.1:9303	Node UUID: e3196d0cd5f1425bf589d91a6105b1c2
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/02 13:46:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:46:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:46:14 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/02 13:46:19 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/02 13:46:24 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/02 13:46:29 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/02 13:46:29 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/02 13:46:29 eventingSetup.go:46: No eventing-con process running
2021/12/02 13:46:29 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/02 13:46:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:46:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9480
2021/12/02 13:46:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9480
2021/12/02 13:46:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:46:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:46:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5629
2021/12/02 13:47:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 629
2021/12/02 13:47:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (378.08s)
=== RUN   TestBucketDeleteAfterPause
2021/12/02 13:47:13 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/02 13:47:13 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/02 13:47:13 eventingSetup.go:46: No eventing-con process running
2021/12/02 13:47:13 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/02 13:47:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:47:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:47:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:47:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:47:27 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:47:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/02 13:47:33 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/02 13:47:38 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/02 13:47:43 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/02 13:47:48 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/02 13:47:48 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/02 13:47:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 13:47:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 851
2021/12/02 13:48:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 851
2021/12/02 13:48:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 851
2021/12/02 13:48:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:48:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:48:07 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/02 13:48:07 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:48:12 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/02 13:48:12 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/02 13:48:12 handler_test.go:716: Deleting source bucket: default
2021/12/02 13:48:19 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/02 13:48:19 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/02 13:48:24 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/02 13:48:24 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/02 13:48:24 eventingSetup.go:46: No eventing-con process running
2021/12/02 13:48:24 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/02 13:48:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:48:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 13:48:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:48:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:48:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2908
2021/12/02 13:48:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 408
2021/12/02 13:48:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (110.01s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/02 13:49:03 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/02 13:49:03 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/02 13:49:03 eventingSetup.go:46: No eventing-con process running
2021/12/02 13:49:03 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/02 13:49:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:49:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:49:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:49:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:49:16 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:49:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/02 13:49:22 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/02 13:49:27 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/02 13:49:32 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/02 13:49:37 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/02 13:49:37 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/02 13:49:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 13:50:12 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/02 13:50:12 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:50:17 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/02 13:50:17 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/02 13:53:17 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/02 13:53:17 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/02 13:53:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/02 13:53:23 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/02 13:53:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:53:23 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/02 13:53:28 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/02 13:53:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:53:28 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2978 expected 5000
2021/12/02 13:53:33 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/02 13:53:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:53:33 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3901 expected 5000
2021/12/02 13:53:38 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/02 13:53:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:53:38 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/02 13:53:38 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/02 13:53:38 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:53:48 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/02 13:53:53 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/02 13:53:58 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/02 13:53:58 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/02 13:53:58 eventingSetup.go:46: No eventing-con process running
2021/12/02 13:53:58 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/02 13:54:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:54:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 13:54:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 13:54:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:54:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:54:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.85s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/02 13:54:32 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/02 13:54:32 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/02 13:54:32 eventingSetup.go:46: No eventing-con process running
2021/12/02 13:54:32 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/02 13:54:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:54:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:54:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:54:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:54:45 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:54:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 13:54:51 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 13:54:56 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 13:55:01 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 13:55:06 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 13:55:06 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/02 13:55:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 13:55:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:55:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2860
2021/12/02 13:55:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2860
2021/12/02 13:55:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:55:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:55:25 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/02 13:55:25 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:55:55 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/02 13:55:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 13:56:01 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 13:56:06 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 13:56:11 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 13:56:16 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 13:56:16 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/02 13:56:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 13:56:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:56:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:56:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:56:34 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:56:34 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/02 13:56:34 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 13:56:44 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/02 13:56:49 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/02 13:56:54 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/02 13:56:54 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/02 13:56:54 eventingSetup.go:46: No eventing-con process running
2021/12/02 13:56:54 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/02 13:56:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:57:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:57:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:57:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.73s)
=== RUN   TestCPPWorkerCleanup
2021/12/02 13:57:17 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/02 13:57:17 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/02 13:57:17 eventingSetup.go:46: No eventing-con process running
2021/12/02 13:57:17 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/02 13:57:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:57:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:57:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:57:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:57:31 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:57:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/02 13:57:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 13:57:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:57:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:57:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:57:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 13:57:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3309
2021/12/02 13:58:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3309
2021/12/02 13:58:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 13:58:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 13:58:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 13:58:09 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/02 13:58:14 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/02 13:58:19 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/02 13:58:24 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/02 13:58:24 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/02 13:58:24 eventingSetup.go:46: No eventing-con process running
2021/12/02 13:58:24 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/02 13:58:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:58:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:58:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:58:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:58:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 13:58:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (129.79s)
=== RUN   TestWithUserXattrs
2021/12/02 13:59:27 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/02 13:59:27 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/02 13:59:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 13:59:27 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/02 13:59:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:59:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:59:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 13:59:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 13:59:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 13:59:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 13:59:44 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/02 13:59:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/02 13:59:50 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/02 13:59:55 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/02 14:00:01 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/02 14:00:06 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/02 14:00:06 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/02 14:00:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 14:00:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:00:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3048
2021/12/02 14:00:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3048
2021/12/02 14:00:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 14:00:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:00:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:00:30 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/02 14:00:35 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/02 14:00:40 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/02 14:00:45 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/02 14:00:45 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/02 14:00:45 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:00:45 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/02 14:00:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:00:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 14:00:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:00:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:01:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3916
2021/12/02 14:01:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1416
2021/12/02 14:01:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (116.11s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/02 14:01:23 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 14:01:23 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/02 14:01:23 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:01:23 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/02 14:01:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:01:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:01:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:01:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:01:36 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:01:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/02 14:01:43 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/02 14:01:48 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/02 14:01:53 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/02 14:01:58 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/02 14:01:58 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/02 14:01:58 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 14:02:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:02:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 976
2021/12/02 14:02:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 976
2021/12/02 14:02:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 14:02:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:02:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:02:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2615
2021/12/02 14:02:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2615
2021/12/02 14:02:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:02:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:02:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 14:02:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:02:47 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 14:03:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 13384
2021/12/02 14:03:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 19633
2021/12/02 14:03:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 24767
2021/12/02 14:03:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/02 14:03:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:03:23 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 14:03:28 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 14:03:33 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 14:03:38 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 14:03:43 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 14:03:43 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/02 14:03:43 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:03:48 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 14:03:48 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/02 14:03:48 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:03:48 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/02 14:03:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:03:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/02 14:04:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/02 14:04:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:04:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:04:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/02 14:04:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (182.92s)
=== RUN   TestMetaBucketDelete
2021/12/02 14:04:26 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/02 14:04:26 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/02 14:04:26 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:04:26 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/02 14:04:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:04:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:04:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:04:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:04:39 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:04:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 14:04:45 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/02 14:04:50 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/02 14:04:55 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/02 14:05:00 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/02 14:05:00 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/02 14:05:00 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/02 14:05:02 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/02 14:05:07 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/02 14:05:07 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/02 14:05:22 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/02 14:05:22 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/02 14:05:22 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:05:22 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/02 14:05:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:05:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:05:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:05:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.76s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/02 14:05:45 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 14:05:45 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 14:05:45 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:05:45 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/02 14:05:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:05:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:05:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:05:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:05:58 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:05:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/02 14:06:18 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/02 14:06:19 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/02 14:06:34 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 14:06:34 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 14:06:49 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 14:06:49 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 14:06:49 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:06:49 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/02 14:06:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:06:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:06:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:07:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (87.36s)
=== RUN   TestSourceBucketDelete
2021/12/02 14:07:12 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/02 14:07:12 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/02 14:07:12 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:07:12 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/02 14:07:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:07:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:07:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:07:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:07:25 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:07:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 14:07:32 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/02 14:07:37 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/02 14:07:42 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/02 14:07:47 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/02 14:07:47 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/02 14:07:47 handler_test.go:945: Deleting source bucket: default
2021/12/02 14:07:48 handler_test.go:947: Deleted source bucket: default
2021/12/02 14:07:53 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/02 14:07:53 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/02 14:08:08 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/02 14:08:08 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/02 14:08:08 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:08:08 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/02 14:08:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:08:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:08:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:08:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.14s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/02 14:08:31 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/02 14:08:31 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/02 14:08:31 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:08:31 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/02 14:08:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:08:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:08:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:08:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:08:45 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:08:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/02 14:09:05 handler_test.go:966: Deleting source bucket: default
2021/12/02 14:09:06 handler_test.go:968: Deleted source bucket: default
2021/12/02 14:09:21 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/02 14:09:21 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/02 14:09:36 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/02 14:09:36 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/02 14:09:36 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:09:36 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/02 14:09:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:09:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:09:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:09:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.81s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/02 14:09:59 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 14:09:59 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 14:09:59 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:09:59 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/02 14:10:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:10:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:10:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:10:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:10:12 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:10:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/02 14:10:32 handler_test.go:988: Deleting source bucket: default
2021/12/02 14:10:34 handler_test.go:990: Deleted source bucket: default
2021/12/02 14:10:34 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/02 14:10:34 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/02 14:10:49 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 14:10:49 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 14:11:04 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 14:11:05 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 14:11:05 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:11:05 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/02 14:11:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:11:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:11:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:11:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.38s)
=== RUN   TestUndeployDuringBootstrap
2021/12/02 14:11:28 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/02 14:11:28 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/02 14:11:28 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:11:28 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/02 14:11:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:11:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:11:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:11:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:11:41 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:11:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/02 14:11:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 14:11:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:11:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:11:54 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/02 14:11:54 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/02 14:11:54 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/02 14:11:59 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/02 14:12:04 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/02 14:12:09 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/02 14:12:14 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/02 14:12:19 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/02 14:12:19 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/02 14:12:19 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:12:44 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/02 14:12:44 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/02 14:12:44 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:12:44 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/02 14:12:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:12:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 14:12:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:12:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:13:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 14:13:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.74s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/02 14:13:12 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:13:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/02 14:13:18 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/02 14:13:23 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/02 14:13:29 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/02 14:13:34 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/02 14:13:34 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/02 14:13:34 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/02 14:13:34 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 14:13:34 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/02 14:13:39 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/02 14:13:39 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/02 14:13:39 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:13:39 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/02 14:13:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:13:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:13:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:13:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.77s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/02 14:14:02 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 14:14:02 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/02 14:14:02 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:14:02 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/02 14:14:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:14:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:14:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:14:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:14:15 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:14:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/02 14:14:22 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/02 14:14:27 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/02 14:14:32 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/02 14:14:37 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/02 14:14:37 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/02 14:14:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 14:15:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:15:12 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 14:15:17 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 14:15:22 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 14:15:27 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 14:15:27 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/02 14:15:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:17:12 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 14:17:12 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/02 14:17:12 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:17:12 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/02 14:17:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:17:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 14:17:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 14:17:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:17:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:17:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3478
2021/12/02 14:17:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3478
2021/12/02 14:17:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (233.10s)
=== RUN   TestUndeployWithKVFailover
2021/12/02 14:17:55 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/02 14:17:55 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/02 14:17:55 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:17:55 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/02 14:17:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:18:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:18:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:18:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:18:08 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:18:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/02 14:18:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 14:18:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:18:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:18:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:18:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:18:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:18:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:18:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1233
2021/12/02 14:18:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3733
2021/12/02 14:18:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 14:18:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 14:18:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 14:18:57 Rebalance progress: 29.16666666666666
2021/12/02 14:19:02 Rebalance progress: 29.16666666666666
2021/12/02 14:19:07 Rebalance progress: 100
2021/12/02 14:19:07 Failing over: [127.0.0.1:19003]
2021/12/02 14:19:17 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 14:20:17 Removing node(s): [] from the cluster
2021/12/02 14:20:22 Rebalance progress: 29.16666666666666
2021/12/02 14:20:27 Rebalance progress: 29.16666666666666
2021/12/02 14:20:32 Rebalance progress: 100
2021/12/02 14:20:37 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/02 14:20:37 App: TestUndeployWithKVFailover got un-deployed
2021/12/02 14:20:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:20:42 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/02 14:20:42 App: TestUndeployWithKVFailover got un-deployed
2021/12/02 14:20:42 No eventing-con process running
2021/12/02 14:20:42 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/02 14:20:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:20:53 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 14:20:58 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 14:20:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:21:00 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:21:10 Waiting for dst bucket item count to get to: 0 curr count: 3510
2021/12/02 14:21:15 Waiting for dst bucket item count to get to: 0 curr count: 3510
2021/12/02 14:21:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (210.04s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/02 14:21:25 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/02 14:21:25 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/02 14:21:25 No eventing-con process running
2021/12/02 14:21:25 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/02 14:21:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:21:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:21:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:21:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:21:38 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:21:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/02 14:21:45 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/02 14:21:50 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/02 14:21:55 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/02 14:22:00 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/02 14:22:00 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/02 14:22:00 Starting bucket ops to source bucket
2021/12/02 14:22:20 Waiting for dst bucket item count to get to: 20000 curr count: 10870
2021/12/02 14:22:25 Waiting for dst bucket item count to get to: 20000 curr count: 16116
2021/12/02 14:22:25 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/02 14:22:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:22:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:22:38 Waiting for dst bucket item count to get to: 0 curr count: 10783
2021/12/02 14:22:43 Waiting for dst bucket item count to get to: 0 curr count: 783
2021/12/02 14:22:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:22:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:22:55 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/02 14:22:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:22:55 Starting bucket ops to source bucket
2021/12/02 14:23:08 Waiting for dst bucket item count to get to: 10000 curr count: 4201
2021/12/02 14:23:13 Waiting for dst bucket item count to get to: 10000 curr count: 4201
2021/12/02 14:23:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 14:23:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:23:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:23:18 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/02 14:23:23 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/02 14:23:28 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/02 14:23:33 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/02 14:23:33 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/02 14:23:33 No eventing-con process running
2021/12/02 14:23:33 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/02 14:23:34 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:23:44 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 14:23:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:23:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:23:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 14:23:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (161.04s)
=== RUN   TestTimerOverwrite
2021/12/02 14:24:06 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/02 14:24:06 App: TestTimerOverwrite got un-deployed
2021/12/02 14:24:06 No eventing-con process running
2021/12/02 14:24:06 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/02 14:24:08 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:24:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:24:15 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:24:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:24:20 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:24:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/02 14:24:26 Waiting for app: TestTimerOverwrite to get deployed
2021/12/02 14:24:31 Waiting for app: TestTimerOverwrite to get deployed
2021/12/02 14:24:36 Waiting for app: TestTimerOverwrite to get deployed
2021/12/02 14:24:41 Waiting for app: TestTimerOverwrite to get deployed
2021/12/02 14:24:41 App: TestTimerOverwrite got deployed
2021/12/02 14:24:51 Starting bucket ops to source bucket
2021/12/02 14:25:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:25:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:25:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:25:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:25:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:25:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:25:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:25:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:25:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:25:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:25:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:25:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 14:26:00 Waiting for dst bucket item count to get to: 5000 curr count: 1800
2021/12/02 14:26:05 Waiting for dst bucket item count to get to: 5000 curr count: 1800
2021/12/02 14:26:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 14:26:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:26:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:26:20 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/02 14:26:25 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/02 14:26:30 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/02 14:26:35 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/02 14:26:35 App: TestTimerOverwrite got un-deployed
2021/12/02 14:26:35 No eventing-con process running
2021/12/02 14:26:35 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/02 14:26:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:26:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 14:26:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:26:48 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:26:58 Waiting for dst bucket item count to get to: 0 curr count: 4489
2021/12/02 14:27:03 Waiting for dst bucket item count to get to: 0 curr count: 1992
2021/12/02 14:27:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (187.19s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/02 14:27:08 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 14:27:09 Removing node(s): [] from the cluster
2021/12/02 14:27:14 Rebalance progress: 100
2021/12/02 14:27:14 Failing over: [127.0.0.1:19003]
2021/12/02 14:27:19 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/02 14:27:19 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/02 14:27:19 No eventing-con process running
2021/12/02 14:27:19 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/02 14:27:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:27:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:27:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:27:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:27:32 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:27:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/02 14:27:33 Starting bucket ops to source bucket
2021/12/02 14:27:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:27:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:27:43 Context: Post to set retry counter, response code: 200 dump: 
2021/12/02 14:27:43 set retry counter, response &{[] }
2021/12/02 14:28:48 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/02 14:28:48 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/02 14:28:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:28:48 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/02 14:28:53 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/02 14:28:53 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/02 14:28:53 No eventing-con process running
2021/12/02 14:28:53 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/02 14:28:55 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:29:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 14:29:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:29:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:29:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 14:29:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:29:16 Removing node(s): [] from the cluster
2021/12/02 14:29:22 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (138.03s)
=== RUN   TestOnUpdateSrcMutation
2021/12/02 14:29:32 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/02 14:29:32 App: TestOnUpdateSrcMutation got un-deployed
2021/12/02 14:29:32 No eventing-con process running
2021/12/02 14:29:32 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/02 14:29:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:29:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:29:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:29:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:29:45 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:29:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/02 14:29:46 Starting bucket ops to source bucket
2021/12/02 14:30:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 14:30:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:30:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:30:18 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/02 14:30:23 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/02 14:30:28 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/02 14:30:33 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/02 14:30:33 App: TestOnUpdateSrcMutation got un-deployed
2021/12/02 14:30:33 No eventing-con process running
2021/12/02 14:30:33 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/02 14:30:35 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:30:45 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 14:30:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:30:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:30:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (89.86s)
=== RUN   TestOnDeleteSrcMutation
2021/12/02 14:31:01 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/02 14:31:01 App: TestOnDeleteSrcMutation got un-deployed
2021/12/02 14:31:01 No eventing-con process running
2021/12/02 14:31:01 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/02 14:31:03 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:31:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:31:10 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:31:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:31:15 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:31:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/02 14:31:16 Starting bucket ops to source bucket
2021/12/02 14:31:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 14:31:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:31:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:31:50 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/02 14:31:55 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/02 14:32:00 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/02 14:32:05 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/02 14:32:05 App: TestOnDeleteSrcMutation got un-deployed
2021/12/02 14:32:05 No eventing-con process running
2021/12/02 14:32:05 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/02 14:32:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:32:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 14:32:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:32:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:32:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (91.56s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/02 14:32:33 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/02 14:32:33 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/02 14:32:33 No eventing-con process running
2021/12/02 14:32:33 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/02 14:32:34 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:32:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:32:41 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:32:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:32:46 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:32:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 14:32:47 Starting bucket ops to source bucket
2021/12/02 14:33:24 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 14:33:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:33:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:33:30 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/02 14:33:35 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/02 14:33:40 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/02 14:33:45 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/02 14:33:45 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/02 14:33:45 No eventing-con process running
2021/12/02 14:33:45 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/02 14:33:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:33:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 14:33:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:33:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:34:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.84s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/02 14:34:13 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/02 14:34:13 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/02 14:34:13 No eventing-con process running
2021/12/02 14:34:13 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/02 14:34:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:34:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:34:21 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:34:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:34:26 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:34:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 14:34:27 Starting bucket ops to source bucket
2021/12/02 14:35:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 14:35:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:35:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:35:11 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/02 14:35:16 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/02 14:35:21 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/02 14:35:26 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/02 14:35:26 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/02 14:35:26 No eventing-con process running
2021/12/02 14:35:26 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/02 14:35:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:35:37 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 14:35:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:35:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:35:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (101.44s)
=== RUN   TestInterHandlerRecursion
2021/12/02 14:35:54 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 14:35:54 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/02 14:35:54 No eventing-con process running
2021/12/02 14:35:54 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/02 14:35:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:36:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:36:02 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:36:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:36:12 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 14:36:12 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/02 14:36:12 No eventing-con process running
2021/12/02 14:36:12 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/02 14:36:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:36:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:36:21 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:36:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:36:26 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:36:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/02 14:36:27 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/02 14:36:32 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 14:36:37 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 14:36:42 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 14:36:47 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 14:36:47 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/02 14:36:47 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:36:47 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/02 14:36:47 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/02 14:36:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:36:52 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 14:36:57 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 14:37:02 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 14:37:07 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 14:37:07 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/02 14:37:07 No eventing-con process running
2021/12/02 14:37:07 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/02 14:37:08 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:37:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:37:15 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:37:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:37:25 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 14:37:25 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/02 14:37:25 No eventing-con process running
2021/12/02 14:37:25 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/02 14:37:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:37:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:37:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:37:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (114.20s)
=== RUN   TestInterBucketRecursion
2021/12/02 14:37:43 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:37:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/02 14:37:45 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/02 14:37:45 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:37:45 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/02 14:37:50 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/02 14:37:55 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/02 14:38:00 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/02 14:38:05 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/02 14:38:05 App: TestInterBucketRecursion_function1 got deployed
2021/12/02 14:38:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:38:10 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/02 14:38:15 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/02 14:38:20 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/02 14:38:25 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/02 14:38:25 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/02 14:38:25 No eventing-con process running
2021/12/02 14:38:25 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/02 14:38:26 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:38:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:38:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:38:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:38:43 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/02 14:38:43 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/02 14:38:43 No eventing-con process running
2021/12/02 14:38:43 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/02 14:38:44 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:38:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:38:51 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:38:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.69s)
=== RUN   TestLargeHandler
2021/12/02 14:38:56 Changed force_compress value to false
2021/12/02 14:38:56 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:38:56 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/02 14:38:56 Changed force_compress value to true
2021/12/02 14:38:56 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:38:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/02 14:39:03 Waiting for app: TestLargeHandler_function to get deployed
2021/12/02 14:39:08 Waiting for app: TestLargeHandler_function to get deployed
2021/12/02 14:39:13 Waiting for app: TestLargeHandler_function to get deployed
2021/12/02 14:39:18 Waiting for app: TestLargeHandler_function to get deployed
2021/12/02 14:39:18 App: TestLargeHandler_function got deployed
2021/12/02 14:39:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:39:23 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/02 14:39:28 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/02 14:39:33 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/02 14:39:38 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/02 14:39:38 App: TestLargeHandler_function got un-deployed
2021/12/02 14:39:38 No eventing-con process running
2021/12/02 14:39:38 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/02 14:39:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:39:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:39:47 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:39:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.45s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/02 14:40:02 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 14:40:02 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/02 14:40:02 No eventing-con process running
2021/12/02 14:40:02 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/02 14:40:03 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:40:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:40:10 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:40:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:40:20 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 14:40:20 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/02 14:40:20 No eventing-con process running
2021/12/02 14:40:20 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/02 14:40:21 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:40:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:40:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:40:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:40:33 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:40:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/02 14:40:34 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/02 14:40:39 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 14:40:44 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 14:40:49 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 14:40:54 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 14:40:54 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/02 14:40:54 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:40:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/02 14:40:56 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/02 14:41:01 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/02 14:41:06 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/02 14:41:11 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/02 14:41:16 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/02 14:41:16 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/02 14:41:16 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/02 14:41:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:41:21 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 14:41:26 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 14:41:31 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 14:41:36 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 14:41:36 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/02 14:41:36 No eventing-con process running
2021/12/02 14:41:36 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/02 14:41:38 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:41:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:41:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:41:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:41:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:41:55 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 14:42:00 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 14:42:05 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 14:42:10 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 14:42:10 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/02 14:42:10 No eventing-con process running
2021/12/02 14:42:10 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/02 14:42:12 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:42:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:42:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:42:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.78s)
=== RUN   TestAllowInterBucketRecursion
2021/12/02 14:42:28 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:42:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/02 14:42:29 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/02 14:42:34 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/02 14:42:40 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/02 14:42:45 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/02 14:42:50 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/02 14:42:50 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/02 14:42:50 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:42:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/02 14:42:51 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/02 14:42:56 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/02 14:43:01 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/02 14:43:06 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/02 14:43:11 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/02 14:43:11 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/02 14:43:11 TestAllowInterBucketRecursion_function2 is deployed
2021/12/02 14:43:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:43:16 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/02 14:43:21 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/02 14:43:26 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/02 14:43:31 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/02 14:43:31 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/02 14:43:31 No eventing-con process running
2021/12/02 14:43:31 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/02 14:43:32 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:43:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:43:41 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:43:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:43:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:43:51 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/02 14:43:56 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/02 14:44:01 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/02 14:44:06 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/02 14:44:06 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/02 14:44:06 No eventing-con process running
2021/12/02 14:44:06 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/02 14:44:07 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:44:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:44:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:44:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.53s)
=== RUN   TestInterBucketRecursion2
2021/12/02 14:44:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:44:31 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:44:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:44:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:44:48 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:44:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/02 14:44:54 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/02 14:44:59 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/02 14:45:04 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/02 14:45:09 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/02 14:45:09 App: TestInterBucketRecursion2_function1 got deployed
2021/12/02 14:45:09 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:45:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/02 14:45:15 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/02 14:45:20 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/02 14:45:25 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/02 14:45:30 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/02 14:45:30 App: TestInterBucketRecursion2_function2 got deployed
2021/12/02 14:45:30 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:45:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/02 14:45:37 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/02 14:45:42 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/02 14:45:47 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/02 14:45:52 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/02 14:45:52 App: TestInterBucketRecursion2_function3 got deployed
2021/12/02 14:45:52 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:45:52 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/02 14:45:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:45:57 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/02 14:46:02 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/02 14:46:07 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/02 14:46:12 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/02 14:46:12 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/02 14:46:12 No eventing-con process running
2021/12/02 14:46:12 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/02 14:46:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:46:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:46:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:46:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:46:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:46:30 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/02 14:46:35 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/02 14:46:40 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/02 14:46:45 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/02 14:46:45 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/02 14:46:45 No eventing-con process running
2021/12/02 14:46:45 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/02 14:46:47 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:46:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:46:54 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:46:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:46:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:47:04 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/02 14:47:09 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/02 14:47:14 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/02 14:47:19 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/02 14:47:19 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/02 14:47:19 No eventing-con process running
2021/12/02 14:47:19 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/02 14:47:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:47:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:47:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:47:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:47:37 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/02 14:47:37 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/02 14:47:37 No eventing-con process running
2021/12/02 14:47:37 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/02 14:47:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:47:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:47:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:47:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.67s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/02 14:47:58 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/02 14:48:03 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/02 14:48:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:48:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:48:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:48:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:49:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:49:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:49:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:49:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:50:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:50:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:50:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:50:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:51:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:51:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:51:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:51:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:52:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:52:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:52:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:52:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:53:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:53:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:53:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 14:53:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:73} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/02 14:54:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:54:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 14:54:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 14:54:11 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/02 14:54:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:54:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 14:54:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 14:54:16 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/02 14:54:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:54:21 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/02 14:54:22 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/02 14:54:22 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:54:22 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/02 14:54:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:54:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:54:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:54:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:54:40 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:54:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 14:54:46 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/02 14:54:51 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/02 14:54:56 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/02 14:55:01 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/02 14:55:01 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: cfa5e75e0e20c14c188e16f4a8d61c49
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 19888b2dca7ad8d8a51006896a74538b
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: e3196d0cd5f1425bf589d91a6105b1c2
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/02 14:55:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:55:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 14:55:06 Rebalance progress: 29.16666666666666
2021/12/02 14:55:11 Rebalance progress: 29.16666666666666
2021/12/02 14:55:16 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: cfa5e75e0e20c14c188e16f4a8d61c49
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 19888b2dca7ad8d8a51006896a74538b
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: e3196d0cd5f1425bf589d91a6105b1c2
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]


2021/12/02 14:55:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:55:16 handler_test.go:1674: Deleting source bucket: default
2021/12/02 14:55:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 14:55:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 14:55:23 Rebalance progress: 29.16666666666666
2021/12/02 14:55:28 Rebalance progress: 29.16666666666666
2021/12/02 14:55:33 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: stopped
	len: 123
	vb list dump: [0, 7-8, 10, 30, 52, 58, 64, 67, 86, 88, 99, 108, 124, 141, 145, 149, 160, 166, 186, 188, 193, 198, 203, 209, 218, 239, 243, 258, 263-264, 279-280, 283, 288, 291, 315, 317, 341, 344, 348, 360, 366, 376, 386, 396, 437, 443, 445, 453-455, 512-513, 517, 527, 534, 538, 540, 548-549, 555, 560, 580, 586, 597, 603, 628, 630, 637, 639, 644, 647-648, 658, 679, 683, 689, 697-698, 701-702, 721, 724, 728-730, 734-735, 744-745, 749, 760, 764, 773, 778, 798, 805, 820, 835, 838, 844, 851, 853-854, 872, 879, 883, 891-892, 898, 939, 947, 955, 961, 982, 990, 996, 1000, 1003, 1006, 1011, 1016]
	stream status: running
	len: 901
	vb list dump: [1-6, 9, 11-29, 31-51, 53-57, 59-63, 65-66, 68-85, 87, 89-98, 100-107, 109-123, 125-140, 142-144, 146-148, 150-159, 161-165, 167-185, 187, 189-192, 194-197, 199-202, 204-208, 210-217, 219-238, 240-242, 244-257, 259-262, 265-278, 281-282, 284-287, 289-290, 292-314, 316, 318-340, 342-343, 345-347, 349-359, 361-365, 367-375, 377-385, 387-395, 397-436, 438-442, 444, 446-452, 456-511, 514-516, 518-526, 528-533, 535-537, 539, 541-547, 550-554, 556-559, 561-579, 581-585, 587-596, 598-602, 604-627, 629, 631-636, 638, 640-643, 645-646, 649-657, 659-678, 680-682, 684-688, 690-696, 699-700, 703-720, 722-723, 725-727, 731-733, 736-743, 746-748, 750-759, 761-763, 765-772, 774-777, 779-797, 799-804, 806-819, 821-834, 836-837, 839-843, 845-850, 852, 855-871, 873-878, 880-882, 884-890, 893-897, 899-938, 940-946, 948-954, 956-960, 962-981, 983-989, 991-995, 997-999, 1001-1002, 1004-1005, 1007-1010, 1012-1015, 1017-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: cfa5e75e0e20c14c188e16f4a8d61c49
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [1-6, 9, 11-29, 31-51, 53-57, 59-63, 65-66, 68-85, 87, 89-98, 100-107, 109-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-123, 125-140, 142-144, 146-148, 150-159, 161-165, 167-185, 187, 189-192, 194-197, 199-202, 204-208, 210-217, 219-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-238, 240-242, 244-257, 259-262, 265-278, 281-282, 284-287, 289-290, 292-314, 316, 318-340]
Producer node: 172.31.2.134:9300	Node UUID: 19888b2dca7ad8d8a51006896a74538b
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-938, 940-946, 948-954, 956-960, 962-981, 983-989, 991-995, 997-999, 1001-1002, 1004-1005, 1007-1010, 1012-1015, 1017-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 94
	v: [684-688, 690-696, 699-700, 703-720, 722-723, 725-727, 731-733, 736-743, 746-748, 750-759, 761-763, 765-772, 774-777, 779-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [797, 799-804, 806-819, 821-834, 836-837, 839-843, 845-850, 852, 855-871, 873-878, 880-882, 884-890, 893-897, 899-910]
Producer node: 127.0.0.1:9303	Node UUID: e3196d0cd5f1425bf589d91a6105b1c2
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [342-343, 345-347, 349-359, 361-365, 367-375, 377-385, 387-395, 397-436, 438-442, 444, 446-452]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 103
	v: [456-511, 514-516, 518-526, 528-533, 535-537, 539, 541-547, 550-554, 556-559, 561-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570-579, 581-585, 587-596, 598-602, 604-627, 629, 631-636, 638, 640-643, 645-646, 649-657, 659-678, 680-682]


2021/12/02 14:55:34 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 14:55:39 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/02 14:55:39 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/02 14:55:44 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/02 14:55:44 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/02 14:55:44 eventingSetup.go:46: No eventing-con process running
2021/12/02 14:55:44 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/02 14:55:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:55:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:55:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:55:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:55:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/02 14:55:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 14:56:02 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (492.43s)
=== RUN   TestN1QLRecursion
2021/12/02 14:56:05 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 14:56:05 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/02 14:56:05 Removing node(s): [] from the cluster
2021/12/02 14:56:10 Rebalance progress: 100
2021/12/02 14:56:16 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:56:22 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:56:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:56:33 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:56:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/02 14:56:35 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/02 14:56:40 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 14:56:45 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 14:56:50 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 14:56:55 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 14:57:00 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 14:57:05 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 14:57:10 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 14:57:15 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 14:57:15 App: TestN1QLRecursion_function1 got deployed
2021/12/02 14:57:15 TestN1QLRecursion_function1 is deployed
2021/12/02 14:57:15 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:57:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/02 14:57:16 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/02 14:57:21 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/02 14:57:26 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/02 14:57:31 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/02 14:57:36 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/02 14:57:36 App: TestN1QLRecursion_function3 got deployed
2021/12/02 14:57:36 TestN1QLRecursion_function3 is deployed
2021/12/02 14:57:36 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:57:36 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/02 14:57:36 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/02 14:57:36 Success: TestN1QLRecursion_function2 is not deployed
2021/12/02 14:57:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:57:41 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/02 14:57:46 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/02 14:57:51 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/02 14:57:56 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/02 14:57:56 App: TestN1QLRecursion_function1 got un-deployed
2021/12/02 14:57:56 No eventing-con process running
2021/12/02 14:57:56 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/02 14:57:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:58:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:58:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:58:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:58:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 14:58:15 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/02 14:58:20 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/02 14:58:25 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/02 14:58:30 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/02 14:58:30 App: TestN1QLRecursion_function3 got un-deployed
2021/12/02 14:58:30 No eventing-con process running
2021/12/02 14:58:30 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/02 14:58:32 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:58:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:58:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:58:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 14:58:45 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 14:58:50 Rebalance progress: 100
--- PASS: TestN1QLRecursion (165.38s)
=== RUN   TestN1QLAllowRecursion
2021/12/02 14:58:50 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 14:58:51 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/02 14:58:51 Removing node(s): [] from the cluster
2021/12/02 14:58:56 Rebalance progress: 100
2021/12/02 14:59:01 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:59:07 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:59:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 14:59:18 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:59:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/02 14:59:19 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/02 14:59:24 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/02 14:59:29 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/02 14:59:34 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/02 14:59:39 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/02 14:59:39 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/02 14:59:39 TestN1QLAllowRecursion_function1 is deployed
2021/12/02 14:59:39 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/02 14:59:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/02 14:59:41 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/02 14:59:46 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/02 14:59:51 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/02 14:59:56 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/02 15:00:01 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/02 15:00:01 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/02 15:00:01 TestN1QLAllowRecursion_function3 is deployed
2021/12/02 15:00:01 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:00:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/02 15:00:02 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/02 15:00:07 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/02 15:00:12 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/02 15:00:17 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/02 15:00:22 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/02 15:00:22 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/02 15:00:22 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/02 15:00:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:00:27 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/02 15:00:32 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/02 15:00:37 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/02 15:00:42 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/02 15:00:42 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/02 15:00:42 No eventing-con process running
2021/12/02 15:00:42 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/02 15:00:44 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:00:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:00:51 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:00:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:00:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:01:01 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/02 15:01:06 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/02 15:01:11 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/02 15:01:16 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/02 15:01:16 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/02 15:01:16 No eventing-con process running
2021/12/02 15:01:16 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/02 15:01:19 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:01:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:01:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:01:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:01:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:01:36 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/02 15:01:41 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/02 15:01:46 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/02 15:01:51 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/02 15:01:51 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/02 15:01:51 No eventing-con process running
2021/12/02 15:01:51 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/02 15:01:53 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:01:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:02:00 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:02:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:02:06 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 15:02:11 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (201.09s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/02 15:02:11 Sleeping for some time
2021/12/02 15:02:21 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/02 15:02:21 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/02 15:02:21 No eventing-con process running
2021/12/02 15:02:21 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/02 15:02:23 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:02:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:02:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:02:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:02:35 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:02:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 15:02:36 Starting bucket ops to source bucket
2021/12/02 15:03:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 15:03:30 Waiting for dst bucket item count to get to: 5000 curr count: 66
2021/12/02 15:03:35 Waiting for dst bucket item count to get to: 5000 curr count: 66
2021/12/02 15:03:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 15:03:35 Starting bucket ops to source bucket
2021/12/02 15:03:37 Starting bucket ops to source bucket
2021/12/02 15:04:42 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/02 15:04:47 Waiting for dst bucket item count to get to: 7000 curr count: 6710
2021/12/02 15:04:52 Waiting for dst bucket item count to get to: 7000 curr count: 6710
2021/12/02 15:04:52 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/02 15:04:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:04:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:04:58 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/02 15:05:03 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/02 15:05:08 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/02 15:05:13 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/02 15:05:13 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/02 15:05:13 No eventing-con process running
2021/12/02 15:05:13 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/02 15:05:15 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:05:25 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 15:05:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:05:26 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:05:36 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/02 15:05:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (204.93s)
=== RUN   TestTimerOverWriteSameReference
2021/12/02 15:05:41 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/02 15:05:41 App: TestTimerOverWriteSameReference got un-deployed
2021/12/02 15:05:41 No eventing-con process running
2021/12/02 15:05:41 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/02 15:05:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:05:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:05:50 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:05:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:05:55 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:05:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/02 15:06:01 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/02 15:06:06 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/02 15:06:11 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/02 15:06:16 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/02 15:06:16 App: TestTimerOverWriteSameReference got deployed
2021/12/02 15:06:26 Starting bucket ops to source bucket
2021/12/02 15:07:28 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/02 15:07:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:07:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:07:43 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/02 15:07:48 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/02 15:07:53 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/02 15:07:58 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/02 15:07:58 App: TestTimerOverWriteSameReference got un-deployed
2021/12/02 15:07:58 No eventing-con process running
2021/12/02 15:07:58 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/02 15:07:59 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:08:09 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/02 15:08:14 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/02 15:08:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:08:16 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:08:26 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/02 15:08:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.88s)
=== RUN   TestCancelTimerBucketop
2021/12/02 15:08:31 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/02 15:08:31 App: TestCancelTimerBucketop got un-deployed
2021/12/02 15:08:31 No eventing-con process running
2021/12/02 15:08:31 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/02 15:08:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:08:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:08:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:08:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:08:45 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:08:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/02 15:08:51 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/02 15:08:56 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/02 15:09:01 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/02 15:09:06 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/02 15:09:06 App: TestCancelTimerBucketop got deployed
2021/12/02 15:09:06 Starting bucket ops to source bucket
2021/12/02 15:09:13 Starting bucket ops to source bucket
2021/12/02 15:10:16 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/02 15:10:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:10:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:10:21 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/02 15:10:26 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/02 15:10:31 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/02 15:10:36 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/02 15:10:36 App: TestCancelTimerBucketop got un-deployed
2021/12/02 15:10:36 No eventing-con process running
2021/12/02 15:10:36 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/02 15:10:38 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:10:48 Waiting for dst bucket item count to get to: 0 curr count: 591
2021/12/02 15:10:53 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/02 15:10:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:10:55 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:11:05 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 15:11:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.38s)
=== RUN   TestJSExpiryDate
2021/12/02 15:11:10 Starting bucket ops to source bucket
2021/12/02 15:11:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 15:11:23 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/02 15:11:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/02 15:11:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:11:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:11:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:11:59 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/02 15:12:04 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/02 15:12:09 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/02 15:12:14 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/02 15:12:14 App: TestJSExpiryDate got un-deployed
2021/12/02 15:12:14 No eventing-con process running
2021/12/02 15:12:14 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/02 15:12:16 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:12:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:12:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:12:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (84.08s)
=== RUN   TestBinaryDoc
2021/12/02 15:12:34 Starting bucket ops to source bucket
2021/12/02 15:12:34 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:12:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/02 15:12:35 Starting bucket ops to source bucket
2021/12/02 15:12:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:12:35 Deleting source bucket docs now
2021/12/02 15:12:35 Starting bucket ops to source bucket
2021/12/02 15:13:12 Waiting for dst bucket item count to get to: 32 curr count: 22
2021/12/02 15:13:17 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/02 15:13:22 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/02 15:13:22 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/02 15:13:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:13:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:13:27 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/02 15:13:33 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/02 15:13:38 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/02 15:13:43 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/02 15:13:43 App: TestBinaryDoc got un-deployed
2021/12/02 15:13:43 No eventing-con process running
2021/12/02 15:13:43 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/02 15:13:44 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:13:54 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/02 15:13:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:13:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:14:06 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/02 15:14:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.56s)
=== RUN   TestConstantBindings
2021/12/02 15:14:06 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:14:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/02 15:14:07 Starting bucket ops to source bucket
2021/12/02 15:14:13 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/02 15:14:18 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/02 15:14:23 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/02 15:14:28 Waiting for dst bucket item count to get to: 6000 curr count: 286
2021/12/02 15:14:33 Waiting for dst bucket item count to get to: 6000 curr count: 286
2021/12/02 15:14:33 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/02 15:14:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:14:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:14:38 Waiting for app: TestConstantBindings to get un-deployed
2021/12/02 15:14:43 Waiting for app: TestConstantBindings to get un-deployed
2021/12/02 15:14:48 Waiting for app: TestConstantBindings to get un-deployed
2021/12/02 15:14:53 Waiting for app: TestConstantBindings to get un-deployed
2021/12/02 15:14:53 App: TestConstantBindings got un-deployed
2021/12/02 15:14:53 No eventing-con process running
2021/12/02 15:14:53 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/02 15:14:55 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:15:05 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 15:15:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:15:07 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:15:17 Waiting for dst bucket item count to get to: 0 curr count: 6000
2021/12/02 15:15:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (70.64s)
=== RUN   TestCodeInsight
2021/12/02 15:15:22 Waiting for app: TestCodeInsight to get un-deployed
2021/12/02 15:15:22 App: TestCodeInsight got un-deployed
2021/12/02 15:15:22 No eventing-con process running
2021/12/02 15:15:22 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/02 15:15:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:15:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:15:31 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:15:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:15:36 Starting bucket ops to source bucket
2021/12/02 15:15:36 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:15:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/02 15:15:42 Waiting for app: TestCodeInsight to get deployed
2021/12/02 15:15:47 Waiting for app: TestCodeInsight to get deployed
2021/12/02 15:15:52 Waiting for app: TestCodeInsight to get deployed
2021/12/02 15:15:57 Waiting for app: TestCodeInsight to get deployed
2021/12/02 15:15:57 App: TestCodeInsight got deployed
2021/12/02 15:16:02 Waiting for dst bucket item count to get to: 1000 curr count: 80
2021/12/02 15:16:02 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/02 15:16:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:16:08 Waiting for app: TestCodeInsight to get un-deployed
2021/12/02 15:16:13 Waiting for app: TestCodeInsight to get un-deployed
2021/12/02 15:16:18 Waiting for app: TestCodeInsight to get un-deployed
2021/12/02 15:16:23 Waiting for app: TestCodeInsight to get un-deployed
2021/12/02 15:16:23 App: TestCodeInsight got un-deployed
2021/12/02 15:16:23 No eventing-con process running
2021/12/02 15:16:23 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/02 15:16:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:16:35 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 15:16:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:16:36 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:16:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 15:16:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (89.47s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.07s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13034.946s

Functional tests took 3 hours, 37 minutes and 19 seconds.

N1ql tests

2021/12/02 15:16:55 Started n1ql
2021/12/02 15:17:05 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/02 15:17:05 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/02 15:17:05 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 15:17:06 Removing node(s): [] from the cluster
2021/12/02 15:17:11 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/02 15:17:16 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/02 15:17:16 App: TestRecursiveMutationN1QL got un-deployed
2021/12/02 15:17:16 No eventing-con process running
2021/12/02 15:17:16 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/02 15:17:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:17:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:17:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:17:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:17:29 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:17:29 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.63s)
=== RUN   TestFlexReset1
2021/12/02 15:17:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/02 15:17:34 App: TestFlexReset1 got un-deployed
2021/12/02 15:17:34 No eventing-con process running
2021/12/02 15:17:34 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/02 15:17:36 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:17:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:17:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:17:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:17:48 Starting bucket ops to source bucket
2021/12/02 15:17:49 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:17:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/02 15:17:55 Waiting for app: TestFlexReset1 to get deployed
2021/12/02 15:18:00 Waiting for app: TestFlexReset1 to get deployed
2021/12/02 15:18:05 Waiting for app: TestFlexReset1 to get deployed
2021/12/02 15:18:10 Waiting for app: TestFlexReset1 to get deployed
2021/12/02 15:18:10 App: TestFlexReset1 got deployed
2021/12/02 15:18:15 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/02 15:18:15 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/02 15:18:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:18:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:18:20 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/02 15:18:25 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/02 15:18:30 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/02 15:18:35 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/02 15:18:35 App: TestFlexReset1 got un-deployed
2021/12/02 15:18:35 No eventing-con process running
2021/12/02 15:18:35 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/02 15:18:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:18:47 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 15:18:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:18:49 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:18:59 Waiting for dst bucket item count to get to: 0 curr count: 971
2021/12/02 15:19:04 Waiting for dst bucket item count to get to: 0 curr count: 471
2021/12/02 15:19:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.43s)
=== RUN   TestFlexReset2
2021/12/02 15:19:09 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/02 15:19:09 App: TestFlexReset2 got un-deployed
2021/12/02 15:19:09 No eventing-con process running
2021/12/02 15:19:09 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/02 15:19:11 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:19:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:19:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:19:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:19:23 Starting bucket ops to source bucket
2021/12/02 15:19:23 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:19:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/02 15:19:29 Waiting for app: TestFlexReset2 to get deployed
2021/12/02 15:19:34 Waiting for app: TestFlexReset2 to get deployed
2021/12/02 15:19:39 Waiting for app: TestFlexReset2 to get deployed
2021/12/02 15:19:44 Waiting for app: TestFlexReset2 to get deployed
2021/12/02 15:19:44 App: TestFlexReset2 got deployed
2021/12/02 15:19:49 Waiting for dst bucket item count to get to: 1000 curr count: 555
2021/12/02 15:19:49 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/02 15:19:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:19:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:19:55 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/02 15:20:00 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/02 15:20:05 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/02 15:20:10 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/02 15:20:10 App: TestFlexReset2 got un-deployed
2021/12/02 15:20:10 No eventing-con process running
2021/12/02 15:20:10 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/02 15:20:11 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:20:21 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 15:20:26 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 15:20:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:20:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:20:38 Waiting for dst bucket item count to get to: 0 curr count: 806
2021/12/02 15:20:43 Waiting for dst bucket item count to get to: 0 curr count: 306
2021/12/02 15:20:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.37s)
=== RUN   TestN1QLLabelledBreak
2021/12/02 15:20:48 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/02 15:20:48 App: TestN1QLLabelledBreak got un-deployed
2021/12/02 15:20:48 No eventing-con process running
2021/12/02 15:20:48 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/02 15:20:50 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:20:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:21:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:21:02 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:21:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/02 15:21:08 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/02 15:21:13 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/02 15:21:18 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/02 15:21:23 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/02 15:21:23 App: TestN1QLLabelledBreak got deployed
2021/12/02 15:21:23 Starting bucket ops to source bucket
2021/12/02 15:21:32 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/02 15:21:37 Waiting for dst bucket item count to get to: 10000 curr count: 285
2021/12/02 15:21:42 Waiting for dst bucket item count to get to: 10000 curr count: 919
2021/12/02 15:21:47 Waiting for dst bucket item count to get to: 10000 curr count: 2881
2021/12/02 15:21:52 Waiting for dst bucket item count to get to: 10000 curr count: 4915
2021/12/02 15:21:57 Waiting for dst bucket item count to get to: 10000 curr count: 6929
2021/12/02 15:22:02 Waiting for dst bucket item count to get to: 10000 curr count: 8934
2021/12/02 15:22:07 Waiting for dst bucket item count to get to: 10000 curr count: 9673
2021/12/02 15:22:07 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 15:22:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:22:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:22:13 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/02 15:22:18 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/02 15:22:23 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/02 15:22:28 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/02 15:22:28 App: TestN1QLLabelledBreak got un-deployed
2021/12/02 15:22:28 No eventing-con process running
2021/12/02 15:22:28 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/02 15:22:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:22:40 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 15:22:45 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 15:22:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:22:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:22:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 15:22:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (133.12s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/02 15:23:01 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/02 15:23:01 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/02 15:23:01 No eventing-con process running
2021/12/02 15:23:01 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/02 15:23:03 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:23:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:23:10 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:23:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:23:15 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:23:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/02 15:23:21 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/02 15:23:26 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/02 15:23:31 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/02 15:23:36 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/02 15:23:36 App: TestN1QLUnlabelledBreak got deployed
2021/12/02 15:23:36 Starting bucket ops to source bucket
2021/12/02 15:23:46 Waiting for dst bucket item count to get to: 10000 curr count: 372
2021/12/02 15:23:51 Waiting for dst bucket item count to get to: 10000 curr count: 372
2021/12/02 15:23:56 Waiting for dst bucket item count to get to: 10000 curr count: 5095
2021/12/02 15:24:01 Waiting for dst bucket item count to get to: 10000 curr count: 5095
2021/12/02 15:24:06 Waiting for dst bucket item count to get to: 10000 curr count: 5095
2021/12/02 15:24:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 15:24:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:24:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:24:11 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/02 15:24:16 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/02 15:24:21 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/02 15:24:26 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/02 15:24:26 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/02 15:24:26 No eventing-con process running
2021/12/02 15:24:26 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/02 15:24:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:24:38 Waiting for dst bucket item count to get to: 0 curr count: 3930
2021/12/02 15:24:43 Waiting for dst bucket item count to get to: 0 curr count: 1430
2021/12/02 15:24:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:24:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:24:55 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 15:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.65s)
=== RUN   TestN1QLThrowStatement
2021/12/02 15:25:00 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/02 15:25:00 App: TestN1QLThrowStatement got un-deployed
2021/12/02 15:25:00 No eventing-con process running
2021/12/02 15:25:00 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/02 15:25:02 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:25:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:25:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:25:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:25:14 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:25:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/02 15:25:20 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/02 15:25:25 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/02 15:25:30 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/02 15:25:35 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/02 15:25:35 App: TestN1QLThrowStatement got deployed
2021/12/02 15:25:35 Starting bucket ops to source bucket
2021/12/02 15:25:44 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/02 15:25:49 Waiting for dst bucket item count to get to: 10000 curr count: 156
2021/12/02 15:25:54 Waiting for dst bucket item count to get to: 10000 curr count: 466
2021/12/02 15:25:59 Waiting for dst bucket item count to get to: 10000 curr count: 4221
2021/12/02 15:26:04 Waiting for dst bucket item count to get to: 10000 curr count: 4221
2021/12/02 15:26:09 Waiting for dst bucket item count to get to: 10000 curr count: 8289
2021/12/02 15:26:14 Waiting for dst bucket item count to get to: 10000 curr count: 8289
2021/12/02 15:26:14 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 15:26:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:26:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:26:19 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/02 15:26:25 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/02 15:26:30 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/02 15:26:35 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/02 15:26:35 App: TestN1QLThrowStatement got un-deployed
2021/12/02 15:26:35 No eventing-con process running
2021/12/02 15:26:35 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/02 15:26:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:26:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 15:26:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:26:48 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:26:58 Waiting for dst bucket item count to get to: 0 curr count: 8447
2021/12/02 15:27:03 Waiting for dst bucket item count to get to: 0 curr count: 3447
2021/12/02 15:27:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (128.43s)
=== RUN   TestN1QLNestedForLoop
2021/12/02 15:27:08 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/02 15:27:08 App: TestN1QLNestedForLoop got un-deployed
2021/12/02 15:27:08 No eventing-con process running
2021/12/02 15:27:08 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/02 15:27:10 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:27:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:27:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:27:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:27:22 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:27:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/02 15:27:28 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/02 15:27:33 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/02 15:27:38 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/02 15:27:43 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/02 15:27:43 App: TestN1QLNestedForLoop got deployed
2021/12/02 15:27:43 Starting bucket ops to source bucket
2021/12/02 15:27:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 15:27:58 Waiting for dst bucket item count to get to: 5000 curr count: 143
2021/12/02 15:28:03 Waiting for dst bucket item count to get to: 5000 curr count: 481
2021/12/02 15:28:08 Waiting for dst bucket item count to get to: 5000 curr count: 1616
2021/12/02 15:28:13 Waiting for dst bucket item count to get to: 5000 curr count: 2820
2021/12/02 15:28:18 Waiting for dst bucket item count to get to: 5000 curr count: 4033
2021/12/02 15:28:23 Waiting for dst bucket item count to get to: 5000 curr count: 4991
2021/12/02 15:28:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 15:28:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:28:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:28:28 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/02 15:28:33 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/02 15:28:38 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/02 15:28:43 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/02 15:28:43 App: TestN1QLNestedForLoop got un-deployed
2021/12/02 15:28:43 No eventing-con process running
2021/12/02 15:28:43 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/02 15:28:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:28:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 15:28:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:28:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:29:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 15:29:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (123.15s)
=== RUN   TestN1QLAutoClose
2021/12/02 15:29:12 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/02 15:29:12 App: TestN1QLAutoClose got un-deployed
2021/12/02 15:29:12 No eventing-con process running
2021/12/02 15:29:12 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/02 15:29:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:29:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:29:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:29:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:29:25 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:29:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/02 15:29:31 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/02 15:29:36 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/02 15:29:41 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/02 15:29:46 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/02 15:29:46 App: TestN1QLAutoClose got deployed
2021/12/02 15:29:46 Starting bucket ops to source bucket
2021/12/02 15:29:52 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/02 15:29:57 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/02 15:30:02 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/02 15:30:07 Waiting for dst bucket item count to get to: 500 curr count: 25
2021/12/02 15:30:12 Waiting for dst bucket item count to get to: 500 curr count: 25
2021/12/02 15:30:17 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/02 15:30:22 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/02 15:30:27 Waiting for dst bucket item count to get to: 500 curr count: 68
2021/12/02 15:30:32 Waiting for dst bucket item count to get to: 500 curr count: 68
2021/12/02 15:30:37 Waiting for dst bucket item count to get to: 500 curr count: 91
2021/12/02 15:30:42 Waiting for dst bucket item count to get to: 500 curr count: 91
2021/12/02 15:30:47 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/02 15:30:52 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/02 15:30:57 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/02 15:31:02 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/02 15:31:07 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/02 15:31:12 Waiting for dst bucket item count to get to: 500 curr count: 155
2021/12/02 15:31:17 Waiting for dst bucket item count to get to: 500 curr count: 166
2021/12/02 15:31:22 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/02 15:31:27 Waiting for dst bucket item count to get to: 500 curr count: 188
2021/12/02 15:31:32 Waiting for dst bucket item count to get to: 500 curr count: 199
2021/12/02 15:31:37 Waiting for dst bucket item count to get to: 500 curr count: 209
2021/12/02 15:31:42 Waiting for dst bucket item count to get to: 500 curr count: 221
2021/12/02 15:31:47 Waiting for dst bucket item count to get to: 500 curr count: 230
2021/12/02 15:31:52 Waiting for dst bucket item count to get to: 500 curr count: 241
2021/12/02 15:31:57 Waiting for dst bucket item count to get to: 500 curr count: 250
2021/12/02 15:32:02 Waiting for dst bucket item count to get to: 500 curr count: 262
2021/12/02 15:32:07 Waiting for dst bucket item count to get to: 500 curr count: 272
2021/12/02 15:32:12 Waiting for dst bucket item count to get to: 500 curr count: 283
2021/12/02 15:32:17 Waiting for dst bucket item count to get to: 500 curr count: 296
2021/12/02 15:32:22 Waiting for dst bucket item count to get to: 500 curr count: 305
2021/12/02 15:32:27 Waiting for dst bucket item count to get to: 500 curr count: 318
2021/12/02 15:32:32 Waiting for dst bucket item count to get to: 500 curr count: 326
2021/12/02 15:32:37 Waiting for dst bucket item count to get to: 500 curr count: 338
2021/12/02 15:32:42 Waiting for dst bucket item count to get to: 500 curr count: 349
2021/12/02 15:32:47 Waiting for dst bucket item count to get to: 500 curr count: 368
2021/12/02 15:32:52 Waiting for dst bucket item count to get to: 500 curr count: 368
2021/12/02 15:32:57 Waiting for dst bucket item count to get to: 500 curr count: 388
2021/12/02 15:33:02 Waiting for dst bucket item count to get to: 500 curr count: 388
2021/12/02 15:33:07 Waiting for dst bucket item count to get to: 500 curr count: 410
2021/12/02 15:33:13 Waiting for dst bucket item count to get to: 500 curr count: 410
2021/12/02 15:33:18 Waiting for dst bucket item count to get to: 500 curr count: 433
2021/12/02 15:33:23 Waiting for dst bucket item count to get to: 500 curr count: 433
2021/12/02 15:33:28 Waiting for dst bucket item count to get to: 500 curr count: 454
2021/12/02 15:33:33 Waiting for dst bucket item count to get to: 500 curr count: 454
2021/12/02 15:33:38 Waiting for dst bucket item count to get to: 500 curr count: 474
2021/12/02 15:33:43 Waiting for dst bucket item count to get to: 500 curr count: 474
2021/12/02 15:33:48 Waiting for dst bucket item count to get to: 500 curr count: 494
2021/12/02 15:33:53 Waiting for dst bucket item count to get to: 500 curr count: 494
2021/12/02 15:33:53 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/02 15:33:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:33:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:33:58 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/02 15:34:03 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/02 15:34:08 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/02 15:34:13 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/02 15:34:13 App: TestN1QLAutoClose got un-deployed
2021/12/02 15:34:13 No eventing-con process running
2021/12/02 15:34:13 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/02 15:34:15 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:34:25 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/02 15:34:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:34:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:34:37 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/02 15:34:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.95s)
=== RUN   TestN1QLPosParams
2021/12/02 15:34:42 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/02 15:34:42 App: TestN1QLPosParams got un-deployed
2021/12/02 15:34:42 No eventing-con process running
2021/12/02 15:34:42 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/02 15:34:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:34:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:34:50 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:34:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:34:55 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:34:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/02 15:35:01 Waiting for app: TestN1QLPosParams to get deployed
2021/12/02 15:35:06 Waiting for app: TestN1QLPosParams to get deployed
2021/12/02 15:35:11 Waiting for app: TestN1QLPosParams to get deployed
2021/12/02 15:35:16 Waiting for app: TestN1QLPosParams to get deployed
2021/12/02 15:35:16 App: TestN1QLPosParams got deployed
2021/12/02 15:35:16 Starting bucket ops to source bucket
2021/12/02 15:35:26 Waiting for dst bucket item count to get to: 5000 curr count: 517
2021/12/02 15:35:31 Waiting for dst bucket item count to get to: 5000 curr count: 517
2021/12/02 15:35:31 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 15:35:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:35:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:35:36 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/02 15:35:41 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/02 15:35:46 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/02 15:35:51 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/02 15:35:51 App: TestN1QLPosParams got un-deployed
2021/12/02 15:35:51 No eventing-con process running
2021/12/02 15:35:51 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/02 15:35:53 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:36:03 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 15:36:08 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 15:36:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:36:10 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:36:20 Waiting for dst bucket item count to get to: 0 curr count: 3005
2021/12/02 15:36:25 Waiting for dst bucket item count to get to: 0 curr count: 3005
2021/12/02 15:36:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.42s)
=== RUN   TestN1QLExhaustConnPool
2021/12/02 15:36:30 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/02 15:36:30 App: TestN1QLExhaustConnPool got un-deployed
2021/12/02 15:36:30 No eventing-con process running
2021/12/02 15:36:30 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/02 15:36:32 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:36:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:36:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:36:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:36:44 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:36:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/02 15:36:50 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/02 15:36:55 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/02 15:37:00 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/02 15:37:05 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/02 15:37:05 App: TestN1QLExhaustConnPool got deployed
2021/12/02 15:37:05 Starting bucket ops to source bucket
2021/12/02 15:37:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 15:37:15 Waiting for dst bucket item count to get to: 100 curr count: 30
2021/12/02 15:37:20 Waiting for dst bucket item count to get to: 100 curr count: 80
2021/12/02 15:37:20 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 15:37:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:37:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:37:25 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/02 15:37:30 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/02 15:37:35 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/02 15:37:40 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/02 15:37:40 App: TestN1QLExhaustConnPool got un-deployed
2021/12/02 15:37:40 No eventing-con process running
2021/12/02 15:37:40 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/02 15:37:42 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:37:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 15:37:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 15:37:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:37:59 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:38:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 15:38:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 15:38:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.99s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/02 15:38:19 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/02 15:38:19 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/02 15:38:19 No eventing-con process running
2021/12/02 15:38:19 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/02 15:38:21 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:38:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:38:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:38:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:38:33 Starting bucket ops to source bucket
2021/12/02 15:38:50 Starting bucket ops to source bucket
2021/12/02 15:39:08 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 13040
2021/12/02 15:39:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/02 15:39:14 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/02 15:39:19 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/02 15:39:24 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/02 15:39:29 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/02 15:39:29 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/02 15:39:29 Starting bucket ops to source bucket
2021/12/02 15:39:35 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 15:39:40 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 15:39:45 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 15:39:50 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 15:39:55 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 15:40:00 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 15:40:05 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 15:40:10 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 15:40:15 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 15:40:20 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 15:40:20 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/02 15:40:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:40:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:40:25 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/02 15:40:30 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/02 15:40:35 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/02 15:40:40 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/02 15:40:40 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/02 15:40:40 No eventing-con process running
2021/12/02 15:40:40 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/02 15:40:42 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:40:52 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/02 15:40:57 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/02 15:40:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:40:59 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:41:09 Waiting for dst bucket item count to get to: 0 curr count: 25525
2021/12/02 15:41:14 Waiting for dst bucket item count to get to: 0 curr count: 10525
2021/12/02 15:41:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.58s)
=== RUN   TestN1QLQueryConsistency
2021/12/02 15:41:19 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/02 15:41:19 App: n1qlquery_consistency got un-deployed
2021/12/02 15:41:19 No eventing-con process running
2021/12/02 15:41:19 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/02 15:41:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:41:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:41:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:41:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:41:32 Starting bucket ops to source bucket
2021/12/02 15:41:35 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:41:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/02 15:41:41 Waiting for app: n1qlquery_consistency to get deployed
2021/12/02 15:41:46 Waiting for app: n1qlquery_consistency to get deployed
2021/12/02 15:41:52 Waiting for app: n1qlquery_consistency to get deployed
2021/12/02 15:41:57 Waiting for app: n1qlquery_consistency to get deployed
2021/12/02 15:41:57 App: n1qlquery_consistency got deployed
2021/12/02 15:42:02 Waiting for dst bucket item count to get to: 5000 curr count: 117
2021/12/02 15:42:07 Waiting for dst bucket item count to get to: 5000 curr count: 2617
2021/12/02 15:42:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 15:42:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 15:42:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:42:12 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/02 15:42:17 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/02 15:42:22 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/02 15:42:27 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/02 15:42:27 App: n1qlquery_consistency got un-deployed
2021/12/02 15:42:27 No eventing-con process running
2021/12/02 15:42:27 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/02 15:42:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:42:39 Waiting for dst bucket item count to get to: 0 curr count: 4360
2021/12/02 15:42:44 Waiting for dst bucket item count to get to: 0 curr count: 1860
2021/12/02 15:42:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:42:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 15:42:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 15:42:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.84s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1555.971s

N1ql tests took 0 hours, 26 minutes and 0 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/02 15:44:30 Started rebalance
=== RUN   TestDuplicateWithEventingRebIn
2021/12/02 15:44:48 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/02 15:44:48 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/02 15:44:48 No eventing-con process running
2021/12/02 15:44:48 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/02 15:44:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:44:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:44:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:44:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:45:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:45:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:45:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:45:07 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:45:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 15:45:13 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/02 15:45:18 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/02 15:45:23 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/02 15:45:28 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/02 15:45:28 App: TestDuplicateWithEventingRebIn got deployed
2021/12/02 15:45:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 15:45:28 Starting bucket ops to source bucket
2021/12/02 15:45:43 Removing node(s): [] from the cluster
2021/12/02 15:45:48 Rebalance progress: 25
2021/12/02 15:45:53 Rebalance progress: 25
2021/12/02 15:45:58 Rebalance progress: 25
2021/12/02 15:46:03 Rebalance progress: 25
2021/12/02 15:46:08 Rebalance progress: 75
2021/12/02 15:46:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:46:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:46:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 15:46:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/02 15:46:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 15:46:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 15:46:18 Rebalance progress: 25
2021/12/02 15:46:23 Rebalance progress: 25
2021/12/02 15:46:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 15:46:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/02 15:46:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 15:46:28 Waiting for src bucket item count to get to: 200000 curr count: 78536
2021/12/02 15:46:33 Waiting for src bucket item count to get to: 200000 curr count: 94688
2021/12/02 15:46:38 Waiting for src bucket item count to get to: 200000 curr count: 94688
2021/12/02 15:46:43 Waiting for src bucket item count to get to: 200000 curr count: 110914
2021/12/02 15:46:48 Waiting for src bucket item count to get to: 200000 curr count: 110914
2021/12/02 15:46:53 Waiting for src bucket item count to get to: 200000 curr count: 110914
2021/12/02 15:46:58 Waiting for src bucket item count to get to: 200000 curr count: 127217
2021/12/02 15:47:03 Waiting for src bucket item count to get to: 200000 curr count: 127217
2021/12/02 15:47:08 Waiting for src bucket item count to get to: 200000 curr count: 143340
2021/12/02 15:47:13 Waiting for src bucket item count to get to: 200000 curr count: 143340
2021/12/02 15:47:18 Waiting for src bucket item count to get to: 200000 curr count: 159655
2021/12/02 15:47:23 Waiting for src bucket item count to get to: 200000 curr count: 159655
2021/12/02 15:47:28 Waiting for src bucket item count to get to: 200000 curr count: 175881
2021/12/02 15:47:33 Waiting for src bucket item count to get to: 200000 curr count: 175881
2021/12/02 15:47:38 Waiting for src bucket item count to get to: 200000 curr count: 191835
2021/12/02 15:47:43 Waiting for src bucket item count to get to: 200000 curr count: 191835
2021/12/02 15:47:53 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 15:47:58 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 15:48:03 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 15:48:08 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 15:48:13 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 15:48:18 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 15:48:24 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 15:48:29 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 15:48:34 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 15:48:39 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 15:48:39 Post SET ops. Expected item count: 200000 got 200001
2021/12/02 15:48:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:48:44 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/02 15:48:49 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/02 15:48:54 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/02 15:48:59 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/02 15:48:59 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/02 15:48:59 No eventing-con process running
2021/12/02 15:48:59 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/02 15:49:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:49:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:49:10 Waiting for dst bucket item count to get to: 0 curr count: 93126
2021/12/02 15:49:15 Waiting for dst bucket item count to get to: 0 curr count: 93126
2021/12/02 15:49:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:49:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:49:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:49:27 Waiting for dst bucket item count to get to: 0 curr count: 200001
2021/12/02 15:49:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (289.09s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/02 15:49:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 15:49:33 Removing node(s): [] from the cluster
2021/12/02 15:49:38 Rebalance progress: 100
2021/12/02 15:49:43 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/02 15:49:43 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/02 15:49:43 No eventing-con process running
2021/12/02 15:49:43 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/02 15:49:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:49:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:49:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:49:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:49:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:49:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:50:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:50:01 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:50:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 15:50:07 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/02 15:50:12 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/02 15:50:17 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/02 15:50:22 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/02 15:50:22 App: TestDuplicateWithEventingRebOut got deployed
2021/12/02 15:50:22 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 15:50:22 Starting bucket ops to source bucket
2021/12/02 15:50:27 Rebalance progress: 25
2021/12/02 15:50:32 Rebalance progress: 25
2021/12/02 15:50:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:50:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:50:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 15:50:37 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/02 15:50:37 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 15:50:37 Waiting for src bucket item count to get to: 200000 curr count: 10637
2021/12/02 15:50:42 Waiting for src bucket item count to get to: 200000 curr count: 26997
2021/12/02 15:50:47 Waiting for src bucket item count to get to: 200000 curr count: 26997
2021/12/02 15:50:52 Waiting for src bucket item count to get to: 200000 curr count: 43549
2021/12/02 15:50:57 Waiting for src bucket item count to get to: 200000 curr count: 43549
2021/12/02 15:51:02 Waiting for src bucket item count to get to: 200000 curr count: 59703
2021/12/02 15:51:07 Waiting for src bucket item count to get to: 200000 curr count: 59703
2021/12/02 15:51:12 Waiting for src bucket item count to get to: 200000 curr count: 59703
2021/12/02 15:51:17 Waiting for src bucket item count to get to: 200000 curr count: 75494
2021/12/02 15:51:22 Waiting for src bucket item count to get to: 200000 curr count: 75494
2021/12/02 15:51:27 Waiting for src bucket item count to get to: 200000 curr count: 91593
2021/12/02 15:51:32 Waiting for src bucket item count to get to: 200000 curr count: 91593
2021/12/02 15:51:37 Waiting for src bucket item count to get to: 200000 curr count: 108160
2021/12/02 15:51:42 Waiting for src bucket item count to get to: 200000 curr count: 108160
2021/12/02 15:51:48 Waiting for src bucket item count to get to: 200000 curr count: 124233
2021/12/02 15:51:53 Waiting for src bucket item count to get to: 200000 curr count: 124233
2021/12/02 15:51:58 Waiting for src bucket item count to get to: 200000 curr count: 140747
2021/12/02 15:52:03 Waiting for src bucket item count to get to: 200000 curr count: 140747
2021/12/02 15:52:08 Waiting for src bucket item count to get to: 200000 curr count: 157254
2021/12/02 15:52:13 Waiting for src bucket item count to get to: 200000 curr count: 157254
2021/12/02 15:52:18 Waiting for src bucket item count to get to: 200000 curr count: 173093
2021/12/02 15:52:23 Waiting for src bucket item count to get to: 200000 curr count: 173093
2021/12/02 15:52:28 Waiting for src bucket item count to get to: 200000 curr count: 189379
2021/12/02 15:52:33 Waiting for src bucket item count to get to: 200000 curr count: 189379
2021/12/02 15:52:38 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 15:52:38 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 15:52:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:52:43 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/02 15:52:48 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/02 15:52:53 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/02 15:52:58 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/02 15:52:58 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/02 15:52:58 No eventing-con process running
2021/12/02 15:52:58 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/02 15:52:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:53:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:53:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:53:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:53:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:53:16 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 15:53:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (229.06s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/02 15:53:26 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 15:53:26 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/02 15:53:26 No eventing-con process running
2021/12/02 15:53:26 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/02 15:53:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:53:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:53:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:53:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:53:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:53:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:53:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:53:45 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:53:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 15:53:51 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/02 15:53:56 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/02 15:54:01 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/02 15:54:06 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/02 15:54:06 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/02 15:54:06 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 15:54:06 Starting bucket ops to source bucket
2021/12/02 15:54:07 Removing node(s): [] from the cluster
2021/12/02 15:54:12 Rebalance progress: 25
2021/12/02 15:54:17 Rebalance progress: 25
2021/12/02 15:54:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:54:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:54:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 15:54:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/02 15:54:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 15:54:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 15:54:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 15:54:35 Rebalance progress: 16.66666666666667
2021/12/02 15:54:40 Rebalance progress: 16.66666666666667
2021/12/02 15:54:45 Rebalance progress: 16.66666666666667
2021/12/02 15:54:50 Rebalance progress: 72.22222222222223
2021/12/02 15:54:55 Rebalance progress: 72.22222222222223
2021/12/02 15:55:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:55:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:55:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 15:55:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 15:55:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 15:55:05 Rebalance progress: 16.66666666666667
2021/12/02 15:55:10 Rebalance progress: 16.66666666666667
2021/12/02 15:55:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:55:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:55:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 15:55:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 15:55:15 Waiting for src bucket item count to get to: 200000 curr count: 94321
2021/12/02 15:55:20 Waiting for src bucket item count to get to: 200000 curr count: 94321
2021/12/02 15:55:25 Waiting for src bucket item count to get to: 200000 curr count: 110265
2021/12/02 15:55:30 Waiting for src bucket item count to get to: 200000 curr count: 110265
2021/12/02 15:55:35 Waiting for src bucket item count to get to: 200000 curr count: 126547
2021/12/02 15:55:40 Waiting for src bucket item count to get to: 200000 curr count: 126547
2021/12/02 15:55:45 Waiting for src bucket item count to get to: 200000 curr count: 142345
2021/12/02 15:55:50 Waiting for src bucket item count to get to: 200000 curr count: 142345
2021/12/02 15:55:55 Waiting for src bucket item count to get to: 200000 curr count: 158468
2021/12/02 15:56:00 Waiting for src bucket item count to get to: 200000 curr count: 158468
2021/12/02 15:56:05 Waiting for src bucket item count to get to: 200000 curr count: 174113
2021/12/02 15:56:10 Waiting for src bucket item count to get to: 200000 curr count: 174113
2021/12/02 15:56:15 Waiting for src bucket item count to get to: 200000 curr count: 189810
2021/12/02 15:56:20 Waiting for src bucket item count to get to: 200000 curr count: 189810
2021/12/02 15:56:25 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 15:56:25 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 15:56:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 15:56:30 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 15:56:35 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 15:56:40 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 15:56:45 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 15:56:45 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/02 15:56:45 No eventing-con process running
2021/12/02 15:56:45 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/02 15:56:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:56:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:56:57 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 15:56:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:56:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:57:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:57:09 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 15:57:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (232.44s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/02 15:57:19 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 15:57:19 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/02 15:57:19 No eventing-con process running
2021/12/02 15:57:19 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/02 15:57:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:57:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:57:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 15:57:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:57:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:57:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:57:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 15:57:37 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 15:57:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 15:57:43 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 15:57:48 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 15:57:53 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 15:57:58 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 15:57:58 App: TestDuplicateWithKVRebIn got deployed
2021/12/02 15:57:58 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 15:57:58 Starting bucket ops to source bucket
2021/12/02 15:57:59 Removing node(s): [] from the cluster
2021/12/02 15:58:04 Rebalance progress: 16.66666666666667
2021/12/02 15:58:09 Rebalance progress: 16.66666666666667
2021/12/02 15:58:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:58:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:58:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 15:58:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 15:58:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 15:58:19 Rebalance progress: 16.66666666666667
2021/12/02 15:58:24 Rebalance progress: 16.66666666666667
2021/12/02 15:58:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:58:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 15:58:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 15:58:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 15:58:29 Waiting for src bucket item count to get to: 200000 curr count: 31143
2021/12/02 15:58:34 Waiting for src bucket item count to get to: 200000 curr count: 47227
2021/12/02 15:58:39 Waiting for src bucket item count to get to: 200000 curr count: 47227
2021/12/02 15:58:44 Waiting for src bucket item count to get to: 200000 curr count: 63460
2021/12/02 15:58:49 Waiting for src bucket item count to get to: 200000 curr count: 63460
2021/12/02 15:58:54 Waiting for src bucket item count to get to: 200000 curr count: 79021
2021/12/02 15:58:59 Waiting for src bucket item count to get to: 200000 curr count: 79021
2021/12/02 15:59:04 Waiting for src bucket item count to get to: 200000 curr count: 79021
2021/12/02 15:59:09 Waiting for src bucket item count to get to: 200000 curr count: 94727
2021/12/02 15:59:14 Waiting for src bucket item count to get to: 200000 curr count: 94727
2021/12/02 15:59:19 Waiting for src bucket item count to get to: 200000 curr count: 110675
2021/12/02 15:59:24 Waiting for src bucket item count to get to: 200000 curr count: 110675
2021/12/02 15:59:29 Waiting for src bucket item count to get to: 200000 curr count: 127106
2021/12/02 15:59:34 Waiting for src bucket item count to get to: 200000 curr count: 127106
2021/12/02 15:59:39 Waiting for src bucket item count to get to: 200000 curr count: 143477
2021/12/02 15:59:44 Waiting for src bucket item count to get to: 200000 curr count: 143477
2021/12/02 15:59:49 Waiting for src bucket item count to get to: 200000 curr count: 159894
2021/12/02 15:59:54 Waiting for src bucket item count to get to: 200000 curr count: 175558
2021/12/02 15:59:59 Waiting for src bucket item count to get to: 200000 curr count: 175558
2021/12/02 16:00:04 Waiting for src bucket item count to get to: 200000 curr count: 191469
2021/12/02 16:00:09 Waiting for src bucket item count to get to: 200000 curr count: 191469
2021/12/02 16:00:14 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 16:00:14 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 16:00:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:00:19 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 16:00:24 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 16:00:29 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 16:00:34 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 16:00:34 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/02 16:00:34 No eventing-con process running
2021/12/02 16:00:34 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/02 16:00:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:00:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:00:46 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 16:00:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:00:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:00:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:00:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 16:00:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (229.07s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/02 16:01:03 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 16:01:03 Removing node(s): [] from the cluster
2021/12/02 16:01:08 Rebalance progress: 100
2021/12/02 16:01:13 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 16:01:13 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/02 16:01:13 No eventing-con process running
2021/12/02 16:01:13 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/02 16:01:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:01:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:01:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:01:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:01:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:01:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:01:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:01:32 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:01:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 16:01:38 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 16:01:43 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 16:01:48 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 16:01:53 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 16:01:53 App: TestDuplicateWithKVRebOut got deployed
2021/12/02 16:01:53 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:01:53 Starting bucket ops to source bucket
2021/12/02 16:01:58 Rebalance progress: 16.66666666666667
2021/12/02 16:02:03 Rebalance progress: 16.66666666666667
2021/12/02 16:02:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:02:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:02:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:02:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 16:02:08 Waiting for src bucket item count to get to: 200000 curr count: 9332
2021/12/02 16:02:13 Waiting for src bucket item count to get to: 200000 curr count: 25365
2021/12/02 16:02:18 Waiting for src bucket item count to get to: 200000 curr count: 25365
2021/12/02 16:02:23 Waiting for src bucket item count to get to: 200000 curr count: 41851
2021/12/02 16:02:28 Waiting for src bucket item count to get to: 200000 curr count: 41851
2021/12/02 16:02:33 Waiting for src bucket item count to get to: 200000 curr count: 41851
2021/12/02 16:02:38 Waiting for src bucket item count to get to: 200000 curr count: 57961
2021/12/02 16:02:43 Waiting for src bucket item count to get to: 200000 curr count: 57961
2021/12/02 16:02:48 Waiting for src bucket item count to get to: 200000 curr count: 73358
2021/12/02 16:02:53 Waiting for src bucket item count to get to: 200000 curr count: 73358
2021/12/02 16:02:58 Waiting for src bucket item count to get to: 200000 curr count: 89143
2021/12/02 16:03:03 Waiting for src bucket item count to get to: 200000 curr count: 89143
2021/12/02 16:03:08 Waiting for src bucket item count to get to: 200000 curr count: 104787
2021/12/02 16:03:13 Waiting for src bucket item count to get to: 200000 curr count: 104787
2021/12/02 16:03:18 Waiting for src bucket item count to get to: 200000 curr count: 120845
2021/12/02 16:03:23 Waiting for src bucket item count to get to: 200000 curr count: 120845
2021/12/02 16:03:28 Waiting for src bucket item count to get to: 200000 curr count: 136800
2021/12/02 16:03:33 Waiting for src bucket item count to get to: 200000 curr count: 136800
2021/12/02 16:03:38 Waiting for src bucket item count to get to: 200000 curr count: 152726
2021/12/02 16:03:43 Waiting for src bucket item count to get to: 200000 curr count: 152726
2021/12/02 16:03:48 Waiting for src bucket item count to get to: 200000 curr count: 168223
2021/12/02 16:03:53 Waiting for src bucket item count to get to: 200000 curr count: 168223
2021/12/02 16:03:58 Waiting for src bucket item count to get to: 200000 curr count: 183823
2021/12/02 16:04:03 Waiting for src bucket item count to get to: 200000 curr count: 183823
2021/12/02 16:04:08 Waiting for src bucket item count to get to: 200000 curr count: 199459
2021/12/02 16:04:13 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 16:04:13 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 16:04:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:04:18 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 16:04:23 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 16:04:28 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 16:04:33 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 16:04:33 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/02 16:04:33 No eventing-con process running
2021/12/02 16:04:33 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/02 16:04:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:04:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:04:45 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 16:04:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:04:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:04:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:04:57 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 16:04:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (239.16s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/02 16:05:07 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 16:05:07 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/02 16:05:07 No eventing-con process running
2021/12/02 16:05:07 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/02 16:05:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:05:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:05:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:05:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:05:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:05:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:05:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:05:26 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:05:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 16:05:32 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 16:05:37 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 16:05:42 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 16:05:47 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 16:05:47 App: TestDuplicateWithKVSwapReb got deployed
2021/12/02 16:05:47 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 16:05:47 Starting bucket ops to source bucket
2021/12/02 16:05:47 Removing node(s): [] from the cluster
2021/12/02 16:05:52 Rebalance progress: 16.66666666666667
2021/12/02 16:05:57 Rebalance progress: 16.66666666666667
2021/12/02 16:06:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:06:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:06:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:06:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 16:06:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 16:06:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:06:08 Rebalance progress: 16.66666666666667
2021/12/02 16:06:13 Rebalance progress: 16.66666666666667
2021/12/02 16:06:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:06:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:06:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:06:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 16:06:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 16:06:23 Rebalance progress: 16.66666666666667
2021/12/02 16:06:28 Rebalance progress: 16.66666666666667
2021/12/02 16:06:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:06:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:06:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:06:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 16:06:33 Waiting for src bucket item count to get to: 200000 curr count: 64990
2021/12/02 16:06:38 Waiting for src bucket item count to get to: 200000 curr count: 64990
2021/12/02 16:06:43 Waiting for src bucket item count to get to: 200000 curr count: 80352
2021/12/02 16:06:48 Waiting for src bucket item count to get to: 200000 curr count: 80352
2021/12/02 16:06:53 Waiting for src bucket item count to get to: 200000 curr count: 96206
2021/12/02 16:06:58 Waiting for src bucket item count to get to: 200000 curr count: 96206
2021/12/02 16:07:03 Waiting for src bucket item count to get to: 200000 curr count: 112595
2021/12/02 16:07:08 Waiting for src bucket item count to get to: 200000 curr count: 112595
2021/12/02 16:07:13 Waiting for src bucket item count to get to: 200000 curr count: 128904
2021/12/02 16:07:18 Waiting for src bucket item count to get to: 200000 curr count: 128904
2021/12/02 16:07:23 Waiting for src bucket item count to get to: 200000 curr count: 145348
2021/12/02 16:07:28 Waiting for src bucket item count to get to: 200000 curr count: 145348
2021/12/02 16:07:33 Waiting for src bucket item count to get to: 200000 curr count: 161662
2021/12/02 16:07:38 Waiting for src bucket item count to get to: 200000 curr count: 161662
2021/12/02 16:07:43 Waiting for src bucket item count to get to: 200000 curr count: 161662
2021/12/02 16:07:48 Waiting for src bucket item count to get to: 200000 curr count: 176760
2021/12/02 16:07:53 Waiting for src bucket item count to get to: 200000 curr count: 176760
2021/12/02 16:07:58 Waiting for src bucket item count to get to: 200000 curr count: 192344
2021/12/02 16:08:03 Waiting for src bucket item count to get to: 200000 curr count: 192344
2021/12/02 16:08:08 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 16:08:08 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 16:08:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:08:13 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 16:08:18 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 16:08:23 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 16:08:28 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 16:08:28 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/02 16:08:28 No eventing-con process running
2021/12/02 16:08:28 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/02 16:08:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:08:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:08:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:08:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:08:47 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 16:08:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.72s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/02 16:08:57 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:08:57 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 16:08:57 No eventing-con process running
2021/12/02 16:08:57 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/02 16:08:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:09:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:09:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:09:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:09:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:09:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:09:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:09:15 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:09:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/02 16:09:21 Starting bucket ops to source bucket
2021/12/02 16:09:26 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 16:09:31 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 16:09:36 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 16:09:36 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:09:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:09:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:09:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 16:09:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:09:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:09:42 Rebalance progress: 16.66666666666667
2021/12/02 16:09:47 Rebalance progress: 16.66666666666667
2021/12/02 16:09:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:09:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:09:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:09:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 16:09:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:09:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:09:57 Rebalance progress: 16.66666666666667
2021/12/02 16:10:02 Rebalance progress: 16.66666666666667
2021/12/02 16:10:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:10:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:10:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:10:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 16:10:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 16:10:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:10:18 Rebalance progress: 12.5
2021/12/02 16:10:23 Rebalance progress: 12.5
2021/12/02 16:10:28 Rebalance progress: 12.5
2021/12/02 16:10:33 Rebalance progress: 70.83333333333334
2021/12/02 16:10:38 Rebalance progress: 70.83333333333334
2021/12/02 16:10:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:10:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:10:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:10:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:10:48 Rebalance progress: 12.5
2021/12/02 16:10:53 Rebalance progress: 12.5
2021/12/02 16:10:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:10:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:10:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:10:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 16:11:03 Rebalance progress: 12.5
2021/12/02 16:11:08 Rebalance progress: 12.5
2021/12/02 16:11:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:11:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:11:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:11:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 16:11:18 Rebalance progress: 12.5
2021/12/02 16:11:23 Rebalance progress: 12.5
2021/12/02 16:11:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:11:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:11:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:11:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:11:33 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:11:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:11:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:11:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:11:48 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 16:11:48 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:11:48 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/02 16:11:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:11:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:11:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:11:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:12:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:12:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12649
2021/12/02 16:12:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.09s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/02 16:12:17 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 16:12:17 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/02 16:12:17 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:12:17 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/02 16:12:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:12:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:12:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:12:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:12:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:12:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:12:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:12:35 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:12:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/02 16:12:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 16:12:47 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/02 16:12:52 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/02 16:12:57 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/02 16:12:57 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:12:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:12:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:12:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:12:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:12:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 16:12:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:13:03 Rebalance progress: 12.5
2021/12/02 16:13:08 Rebalance progress: 12.5
2021/12/02 16:13:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:13:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:13:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:13:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/02 16:13:18 Rebalance progress: 12.5
2021/12/02 16:13:23 Rebalance progress: 12.5
2021/12/02 16:13:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:13:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:13:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:13:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:13:33 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 16:13:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 16:13:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 16:13:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 16:13:48 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/02 16:13:48 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:13:48 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/02 16:13:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:13:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:13:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:13:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:14:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:14:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4596
2021/12/02 16:14:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (119.62s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/02 16:14:16 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 16:14:16 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/02 16:14:16 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:14:16 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/02 16:14:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:14:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:14:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:14:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:14:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:14:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:14:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:14:35 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:14:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 16:14:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 16:14:46 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/02 16:14:51 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/02 16:14:56 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/02 16:14:56 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:14:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:14:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:14:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:14:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:15:02 Rebalance progress: 12.5
2021/12/02 16:15:07 Rebalance progress: 12.5
2021/12/02 16:15:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:15:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:15:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:15:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:15:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:15:17 Rebalance progress: 12.5
2021/12/02 16:15:22 Rebalance progress: 12.5
2021/12/02 16:15:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:15:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:15:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:15:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 16:15:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:15:33 Rebalance progress: 12.5
2021/12/02 16:15:38 Rebalance progress: 12.5
2021/12/02 16:15:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:15:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:15:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:15:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:15:48 Rebalance progress: 12.5
2021/12/02 16:15:53 Rebalance progress: 12.5
2021/12/02 16:15:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:15:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:15:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:15:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 16:16:03 Rebalance progress: 12.5
2021/12/02 16:16:08 Rebalance progress: 12.5
2021/12/02 16:16:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:16:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:16:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:16:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 16:16:18 Rebalance progress: 12.5
2021/12/02 16:16:23 Rebalance progress: 12.5
2021/12/02 16:16:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:16:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:16:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:16:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:16:33 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 16:16:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 16:16:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 16:16:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 16:16:48 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/02 16:16:48 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:16:48 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/02 16:16:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:16:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:17:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37
2021/12/02 16:17:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37
2021/12/02 16:17:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:17:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:17:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:17:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10656
2021/12/02 16:17:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.36s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/02 16:17:27 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 16:17:27 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/02 16:17:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:17:27 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/02 16:17:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:17:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:17:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:17:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:17:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:17:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:17:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:17:40 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:17:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 16:17:46 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 16:17:51 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/02 16:17:56 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/02 16:18:01 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/02 16:18:02 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:18:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:18:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:18:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:18:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:18:07 Rebalance progress: 12.5
2021/12/02 16:18:12 Rebalance progress: 12.5
2021/12/02 16:18:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:18:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:18:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:18:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:18:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:18:23 Rebalance progress: 12.5
2021/12/02 16:18:28 Rebalance progress: 12.5
2021/12/02 16:18:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:18:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:18:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:18:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 16:18:38 Rebalance progress: 12.5
2021/12/02 16:18:43 Rebalance progress: 12.5
2021/12/02 16:18:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:18:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:18:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:18:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:18:53 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 16:18:58 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 16:19:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 16:19:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 16:19:08 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/02 16:19:08 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:19:08 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/02 16:19:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:19:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:19:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:19:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:19:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:19:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6121
2021/12/02 16:19:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (129.79s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/02 16:19:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:19:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:19:37 Rebalance progress: 100
2021/12/02 16:19:42 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 16:19:42 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/02 16:19:42 No eventing-con process running
2021/12/02 16:19:42 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/02 16:19:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:19:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:19:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:19:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:19:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:19:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:19:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:19:56 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:19:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/02 16:20:02 Starting bucket ops to source bucket
2021/12/02 16:20:07 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/02 16:20:12 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/02 16:20:17 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/02 16:20:17 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:20:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:20:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:20:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:20:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:20:22 Rebalance progress: 12.5
2021/12/02 16:20:27 Rebalance progress: 12.5
2021/12/02 16:20:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:20:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:20:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:20:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/02 16:20:32 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/02 16:20:37 Rebalance progress: 12.5
2021/12/02 16:20:42 Rebalance progress: 12.5
2021/12/02 16:20:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:20:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:20:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:20:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:20:53 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 16:20:58 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 16:21:03 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 16:21:08 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 16:21:08 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/02 16:21:08 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:21:08 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/02 16:21:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:21:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:21:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:21:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:21:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:21:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4540
2021/12/02 16:21:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (119.75s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/02 16:21:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:21:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:21:37 Rebalance progress: 100
2021/12/02 16:21:42 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 16:21:42 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/02 16:21:42 No eventing-con process running
2021/12/02 16:21:42 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/02 16:21:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:21:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:21:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:21:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:21:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:21:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:21:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:21:55 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:21:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/02 16:22:01 Starting bucket ops to source bucket
2021/12/02 16:22:06 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/02 16:22:11 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/02 16:22:16 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/02 16:22:16 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:22:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:22:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:22:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:22:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:22:22 Rebalance progress: 12.5
2021/12/02 16:22:27 Rebalance progress: 12.5
2021/12/02 16:22:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:22:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:22:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:22:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/02 16:22:32 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/02 16:22:37 Rebalance progress: 12.5
2021/12/02 16:22:42 Rebalance progress: 12.5
2021/12/02 16:22:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:22:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:22:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:22:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:22:47 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/02 16:22:52 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 16:22:57 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 16:23:02 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 16:23:07 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 16:23:07 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/02 16:23:07 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:23:07 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/02 16:23:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:23:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:23:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37471
2021/12/02 16:23:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:23:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:23:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:23:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37471
2021/12/02 16:23:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37471
2021/12/02 16:23:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.78s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/02 16:23:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 16:23:46 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/02 16:23:46 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:23:46 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/02 16:23:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:23:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:23:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:23:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:24:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:24:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:24:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:24:05 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:24:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:24:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:24:06 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:24:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 16:24:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 16:24:17 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/02 16:24:22 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/02 16:24:27 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/02 16:24:27 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/02 16:24:32 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/02 16:24:37 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/02 16:24:42 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/02 16:24:42 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:24:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:24:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:24:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:24:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:24:48 Rebalance progress: 12.5
2021/12/02 16:24:53 Rebalance progress: 12.5
2021/12/02 16:24:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:24:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:24:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:24:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:24:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:25:03 Rebalance progress: 12.5
2021/12/02 16:25:08 Rebalance progress: 12.5
2021/12/02 16:25:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:25:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:25:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:25:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 16:25:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:25:18 Rebalance progress: 12.5
2021/12/02 16:25:23 Rebalance progress: 12.5
2021/12/02 16:25:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:25:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:25:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:25:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:25:34 Rebalance progress: 12.5
2021/12/02 16:25:39 Rebalance progress: 12.5
2021/12/02 16:25:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:25:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:25:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:25:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 16:25:49 Rebalance progress: 12.5
2021/12/02 16:25:54 Rebalance progress: 12.5
2021/12/02 16:25:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:25:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:25:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:25:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 16:26:04 Rebalance progress: 12.5
2021/12/02 16:26:09 Rebalance progress: 12.5
2021/12/02 16:26:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:26:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:26:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:26:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:26:19 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 16:26:24 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 16:26:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 16:26:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 16:26:34 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/02 16:26:34 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:26:34 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/02 16:26:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:26:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:26:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12169
2021/12/02 16:26:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:26:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:26:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:26:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:26:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:27:00 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 16:27:05 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 16:27:10 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 16:27:15 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 16:27:15 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/02 16:27:15 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:27:15 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/02 16:27:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:27:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:27:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:27:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:27:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:27:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (232.32s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/02 16:27:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 16:27:38 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/02 16:27:38 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:27:38 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/02 16:27:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:27:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:27:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:27:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:27:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:27:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:27:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:27:57 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:27:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:27:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:27:58 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:27:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 16:28:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 16:28:09 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/02 16:28:14 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/02 16:28:19 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/02 16:28:19 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/02 16:28:24 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/02 16:28:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/02 16:28:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/02 16:28:34 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:28:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:28:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:28:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:28:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:28:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 16:28:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:28:41 Rebalance progress: 12.5
2021/12/02 16:28:46 Rebalance progress: 12.5
2021/12/02 16:28:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:28:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:28:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:28:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/02 16:28:56 Rebalance progress: 12.5
2021/12/02 16:29:01 Rebalance progress: 12.5
2021/12/02 16:29:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:29:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:29:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:29:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:29:11 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 16:29:16 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 16:29:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 16:29:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 16:29:26 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/02 16:29:26 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:29:26 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/02 16:29:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:29:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:29:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:29:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:29:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:29:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6158
2021/12/02 16:29:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:29:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:29:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 16:29:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 16:30:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 16:30:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 16:30:07 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/02 16:30:07 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:30:07 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/02 16:30:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:30:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:30:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:30:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:30:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:30:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (172.56s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/02 16:30:31 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 16:30:31 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/02 16:30:31 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:30:31 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/02 16:30:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:30:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:30:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:30:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:30:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:30:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:30:50 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 16:30:50 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/02 16:30:50 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:30:50 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/02 16:30:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:30:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:30:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:30:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:31:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:31:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:31:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:31:03 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:31:04 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:31:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:31:04 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:31:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/02 16:31:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 16:31:15 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/02 16:31:21 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/02 16:31:26 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/02 16:31:26 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/02 16:31:31 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/02 16:31:36 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/02 16:31:41 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/02 16:31:41 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:31:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:31:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:31:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:31:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:31:46 Rebalance progress: 12.5
2021/12/02 16:31:51 Rebalance progress: 12.5
2021/12/02 16:31:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:31:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:31:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:31:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:31:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:32:02 Rebalance progress: 12.5
2021/12/02 16:32:07 Rebalance progress: 12.5
2021/12/02 16:32:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:32:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:32:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:32:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 16:32:17 Rebalance progress: 12.5
2021/12/02 16:32:22 Rebalance progress: 12.5
2021/12/02 16:32:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:32:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:32:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:32:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:32:32 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 16:32:37 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 16:32:42 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 16:32:47 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 16:32:47 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/02 16:32:47 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:32:47 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/02 16:32:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:32:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:33:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 629
2021/12/02 16:33:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 629
2021/12/02 16:33:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:33:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:33:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:33:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7638
2021/12/02 16:33:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:33:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:33:23 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 16:33:28 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 16:33:33 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 16:33:38 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 16:33:38 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/02 16:33:38 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:33:38 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/02 16:33:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:33:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:33:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:33:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:33:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:33:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.70s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/02 16:34:02 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:34:02 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 16:34:02 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:34:02 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/02 16:34:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:34:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:34:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:34:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:34:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:34:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:34:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:34:20 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:34:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/02 16:34:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 16:34:31 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 16:34:36 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 16:34:41 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 16:34:41 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:34:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:34:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:34:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:34:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:35:02 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:35:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:35:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:35:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:35:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:35:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:35:38 Rebalance progress: 12.5
2021/12/02 16:35:43 Rebalance progress: 12.5
2021/12/02 16:35:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:35:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:35:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:35:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:35:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:35:53 Rebalance progress: 12.5
2021/12/02 16:35:58 Rebalance progress: 12.5
2021/12/02 16:36:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:36:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:36:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:36:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 16:36:08 Rebalance progress: 12.5
2021/12/02 16:36:13 Rebalance progress: 12.5
2021/12/02 16:36:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:36:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:36:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:36:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:36:23 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:36:28 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:36:33 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:36:38 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:36:38 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 16:36:38 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:36:38 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/02 16:36:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:36:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:36:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1695
2021/12/02 16:36:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1695
2021/12/02 16:36:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:36:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:37:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:37:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11161
2021/12/02 16:37:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.20s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/02 16:37:17 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:37:17 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 16:37:17 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:37:17 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/02 16:37:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:37:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:37:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:37:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:37:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:37:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:37:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:37:35 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:37:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/02 16:37:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 16:37:47 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 16:37:52 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 16:37:57 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 16:37:57 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:37:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:37:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:37:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:37:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:38:17 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:38:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:38:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:38:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:38:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:38:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:38:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:38:48 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:38:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:38:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:38:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:38:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:38:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:38:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:39:19 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:39:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:39:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:39:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:39:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:39:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:39:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:39:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:39:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:39:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:40:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:40:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:40:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:40:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:40:20 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:40:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:40:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:40:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:40:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:40:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:40:36 Rebalance progress: 12.5
2021/12/02 16:40:41 Rebalance progress: 12.5
2021/12/02 16:40:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:40:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:40:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:40:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:40:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:40:51 Rebalance progress: 12.5
2021/12/02 16:40:56 Rebalance progress: 12.5
2021/12/02 16:41:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:41:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:41:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:41:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 16:41:07 Rebalance progress: 12.5
2021/12/02 16:41:12 Rebalance progress: 12.5
2021/12/02 16:41:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:41:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:41:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:41:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:41:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:41:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:41:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:41:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 16:41:37 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 16:41:37 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:41:37 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/02 16:41:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:41:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:41:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21483
2021/12/02 16:41:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:41:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:41:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:42:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9169
2021/12/02 16:42:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9169
2021/12/02 16:42:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (298.51s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/02 16:42:15 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 16:42:15 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/02 16:42:15 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:42:15 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/02 16:42:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:42:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:42:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:42:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:42:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:42:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:42:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:42:34 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:42:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 16:42:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 16:42:45 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/02 16:42:50 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/02 16:42:55 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/02 16:42:55 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:42:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:42:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:42:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:42:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:43:01 Rebalance progress: 12.5
2021/12/02 16:43:06 Rebalance progress: 12.5
2021/12/02 16:43:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:43:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:43:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:43:11 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:43:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:43:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:43:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:43:26 Rebalance progress: 12.5
2021/12/02 16:43:31 Rebalance progress: 12.5
2021/12/02 16:43:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:43:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:43:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:43:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 16:43:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:43:41 Rebalance progress: 12.5
2021/12/02 16:43:46 Rebalance progress: 12.5
2021/12/02 16:43:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:43:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:43:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:43:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 16:43:57 Rebalance progress: 12.5
2021/12/02 16:44:02 Rebalance progress: 12.5
2021/12/02 16:44:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:44:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:44:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:44:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:44:12 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 16:44:17 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 16:44:22 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 16:44:27 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 16:44:27 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/02 16:44:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:44:27 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/02 16:44:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:44:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:44:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8638
2021/12/02 16:44:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:44:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:44:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:44:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3589
2021/12/02 16:44:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3589
2021/12/02 16:44:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.95s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/02 16:45:05 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 16:45:05 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/02 16:45:05 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:45:05 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/02 16:45:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:45:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:45:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:45:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:45:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:45:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:45:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:45:24 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:45:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 16:45:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 16:45:35 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/02 16:45:40 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/02 16:45:45 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/02 16:45:45 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:45:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:45:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:45:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/02 16:45:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:45:50 Rebalance progress: 12.5
2021/12/02 16:45:55 Rebalance progress: 12.5
2021/12/02 16:46:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:46:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:46:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:46:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/02 16:46:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:46:06 Rebalance progress: 12.5
2021/12/02 16:46:11 Rebalance progress: 12.5
2021/12/02 16:46:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:46:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:46:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:46:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 16:46:21 Rebalance progress: 12.5
2021/12/02 16:46:26 Rebalance progress: 12.5
2021/12/02 16:46:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:46:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:46:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:46:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:46:36 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 16:46:41 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 16:46:46 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 16:46:51 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 16:46:51 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/02 16:46:51 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:46:51 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/02 16:46:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:46:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:47:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6053
2021/12/02 16:47:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6053
2021/12/02 16:47:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:47:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:47:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:47:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.16s)
=== RUN   TestEventingRebInWithOps
2021/12/02 16:47:24 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 16:47:24 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/02 16:47:25 eventingSetup.go:46: No eventing-con process running
2021/12/02 16:47:25 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/02 16:47:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:47:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:47:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:47:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:47:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:47:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:47:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:47:43 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:47:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 16:47:49 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 16:47:54 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 16:47:59 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 16:48:04 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 16:48:04 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/02 16:48:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:48:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 16:48:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 16:48:10 Rebalance progress: 12.5
2021/12/02 16:48:15 Rebalance progress: 12.5
2021/12/02 16:48:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:48:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:48:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:48:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:48:25 Rebalance progress: 12.5
2021/12/02 16:48:30 Rebalance progress: 12.5
2021/12/02 16:48:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:48:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:48:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:48:35 Waiting for src bucket item count to get to: 200000 curr count: 37072
2021/12/02 16:48:40 Waiting for src bucket item count to get to: 200000 curr count: 37072
2021/12/02 16:48:45 Waiting for src bucket item count to get to: 200000 curr count: 52629
2021/12/02 16:48:50 Waiting for src bucket item count to get to: 200000 curr count: 52629
2021/12/02 16:48:55 Waiting for src bucket item count to get to: 200000 curr count: 68634
2021/12/02 16:49:00 Waiting for src bucket item count to get to: 200000 curr count: 83711
2021/12/02 16:49:05 Waiting for src bucket item count to get to: 200000 curr count: 83711
2021/12/02 16:49:10 Waiting for src bucket item count to get to: 200000 curr count: 99645
2021/12/02 16:49:15 Waiting for src bucket item count to get to: 200000 curr count: 99645
2021/12/02 16:49:20 Waiting for src bucket item count to get to: 200000 curr count: 99645
2021/12/02 16:49:25 Waiting for src bucket item count to get to: 200000 curr count: 115448
2021/12/02 16:49:30 Waiting for src bucket item count to get to: 200000 curr count: 115448
2021/12/02 16:49:35 Waiting for src bucket item count to get to: 200000 curr count: 131253
2021/12/02 16:49:40 Waiting for src bucket item count to get to: 200000 curr count: 131253
2021/12/02 16:49:45 Waiting for src bucket item count to get to: 200000 curr count: 146911
2021/12/02 16:49:50 Waiting for src bucket item count to get to: 200000 curr count: 146911
2021/12/02 16:49:55 Waiting for src bucket item count to get to: 200000 curr count: 162698
2021/12/02 16:50:00 Waiting for src bucket item count to get to: 200000 curr count: 162698
2021/12/02 16:50:05 Waiting for src bucket item count to get to: 200000 curr count: 177422
2021/12/02 16:50:10 Waiting for src bucket item count to get to: 200000 curr count: 177422
2021/12/02 16:50:15 Waiting for src bucket item count to get to: 200000 curr count: 193217
2021/12/02 16:50:20 Waiting for src bucket item count to get to: 200000 curr count: 193217
2021/12/02 16:50:25 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 16:50:25 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 16:50:25 Starting bucket ops to source bucket
2021/12/02 16:54:17 Waiting for dst bucket item count to get to: 0 curr count: 4923
2021/12/02 16:54:22 Waiting for dst bucket item count to get to: 0 curr count: 4923
2021/12/02 16:54:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:54:22 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 16:54:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 16:54:27 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 16:54:32 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 16:54:37 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 16:54:42 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 16:54:42 App: TestEventingRebInWithOps got un-deployed
2021/12/02 16:54:42 No eventing-con process running
2021/12/02 16:54:42 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/02 16:54:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:54:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:54:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:54:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:54:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:54:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (461.35s)
=== RUN   TestEventingRebOutWithOps
2021/12/02 16:55:01 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 16:55:01 Removing node(s): [] from the cluster
2021/12/02 16:55:06 Rebalance progress: 100
2021/12/02 16:55:11 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 16:55:11 App: TestEventingRebOutWithOps got un-deployed
2021/12/02 16:55:11 No eventing-con process running
2021/12/02 16:55:11 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/02 16:55:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:55:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:55:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 16:55:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:55:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:55:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:55:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 16:55:30 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 16:55:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 16:55:36 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 16:55:41 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 16:55:46 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 16:55:51 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 16:55:51 App: TestEventingRebOutWithOps got deployed
2021/12/02 16:55:51 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 16:55:51 Starting bucket ops to source bucket
2021/12/02 16:55:56 Rebalance progress: 12.5
2021/12/02 16:56:01 Rebalance progress: 12.5
2021/12/02 16:56:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:56:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 16:56:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 16:56:06 Waiting for src bucket item count to get to: 200000 curr count: 11821
2021/12/02 16:56:11 Waiting for src bucket item count to get to: 200000 curr count: 27617
2021/12/02 16:56:16 Waiting for src bucket item count to get to: 200000 curr count: 27617
2021/12/02 16:56:21 Waiting for src bucket item count to get to: 200000 curr count: 43418
2021/12/02 16:56:26 Waiting for src bucket item count to get to: 200000 curr count: 43418
2021/12/02 16:56:31 Waiting for src bucket item count to get to: 200000 curr count: 59231
2021/12/02 16:56:36 Waiting for src bucket item count to get to: 200000 curr count: 59231
2021/12/02 16:56:41 Waiting for src bucket item count to get to: 200000 curr count: 74187
2021/12/02 16:56:46 Waiting for src bucket item count to get to: 200000 curr count: 74187
2021/12/02 16:56:51 Waiting for src bucket item count to get to: 200000 curr count: 74187
2021/12/02 16:56:56 Waiting for src bucket item count to get to: 200000 curr count: 89459
2021/12/02 16:57:01 Waiting for src bucket item count to get to: 200000 curr count: 89459
2021/12/02 16:57:06 Waiting for src bucket item count to get to: 200000 curr count: 105151
2021/12/02 16:57:11 Waiting for src bucket item count to get to: 200000 curr count: 105151
2021/12/02 16:57:16 Waiting for src bucket item count to get to: 200000 curr count: 120993
2021/12/02 16:57:21 Waiting for src bucket item count to get to: 200000 curr count: 120993
2021/12/02 16:57:26 Waiting for src bucket item count to get to: 200000 curr count: 136781
2021/12/02 16:57:31 Waiting for src bucket item count to get to: 200000 curr count: 136781
2021/12/02 16:57:36 Waiting for src bucket item count to get to: 200000 curr count: 152763
2021/12/02 16:57:42 Waiting for src bucket item count to get to: 200000 curr count: 152763
2021/12/02 16:57:47 Waiting for src bucket item count to get to: 200000 curr count: 167586
2021/12/02 16:57:52 Waiting for src bucket item count to get to: 200000 curr count: 167586
2021/12/02 16:57:57 Waiting for src bucket item count to get to: 200000 curr count: 183048
2021/12/02 16:58:02 Waiting for src bucket item count to get to: 200000 curr count: 183048
2021/12/02 16:58:07 Waiting for src bucket item count to get to: 200000 curr count: 198842
2021/12/02 16:58:12 Waiting for src bucket item count to get to: 200000 curr count: 198842
2021/12/02 16:58:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 16:58:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 16:58:17 Starting bucket ops to source bucket
2021/12/02 17:02:08 Waiting for dst bucket item count to get to: 0 curr count: 5809
2021/12/02 17:02:13 Waiting for dst bucket item count to get to: 0 curr count: 5809
2021/12/02 17:02:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:02:13 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 17:02:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 17:02:18 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 17:02:23 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 17:02:28 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 17:02:33 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 17:02:33 App: TestEventingRebOutWithOps got un-deployed
2021/12/02 17:02:33 No eventing-con process running
2021/12/02 17:02:33 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/02 17:02:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:02:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:02:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:02:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:02:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:02:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (470.49s)
=== RUN   TestEventingSwapRebWithOps
2021/12/02 17:02:56 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 17:02:56 App: TestEventingSwapRebWithOps got un-deployed
2021/12/02 17:02:56 No eventing-con process running
2021/12/02 17:02:56 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/02 17:02:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:03:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:03:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:03:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:03:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:03:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:03:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:03:15 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 17:03:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 17:03:21 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 17:03:26 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 17:03:31 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 17:03:36 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 17:03:36 App: TestEventingSwapRebWithOps got deployed
2021/12/02 17:03:36 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 17:03:36 Starting bucket ops to source bucket
2021/12/02 17:03:37 Removing node(s): [] from the cluster
2021/12/02 17:03:42 Rebalance progress: 12.5
2021/12/02 17:03:47 Rebalance progress: 12.5
2021/12/02 17:03:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:03:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:03:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:03:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 17:03:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 17:03:57 Rebalance progress: 12.5
2021/12/02 17:04:02 Rebalance progress: 12.5
2021/12/02 17:04:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:04:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:04:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:04:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 17:04:12 Rebalance progress: 12.5
2021/12/02 17:04:17 Rebalance progress: 12.5
2021/12/02 17:04:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:04:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:04:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:04:22 Waiting for src bucket item count to get to: 200000 curr count: 64973
2021/12/02 17:04:27 Waiting for src bucket item count to get to: 200000 curr count: 64973
2021/12/02 17:04:32 Waiting for src bucket item count to get to: 200000 curr count: 79345
2021/12/02 17:04:37 Waiting for src bucket item count to get to: 200000 curr count: 79345
2021/12/02 17:04:42 Waiting for src bucket item count to get to: 200000 curr count: 95076
2021/12/02 17:04:47 Waiting for src bucket item count to get to: 200000 curr count: 95076
2021/12/02 17:04:52 Waiting for src bucket item count to get to: 200000 curr count: 111177
2021/12/02 17:04:57 Waiting for src bucket item count to get to: 200000 curr count: 111177
2021/12/02 17:05:02 Waiting for src bucket item count to get to: 200000 curr count: 127091
2021/12/02 17:05:07 Waiting for src bucket item count to get to: 200000 curr count: 127091
2021/12/02 17:05:12 Waiting for src bucket item count to get to: 200000 curr count: 143057
2021/12/02 17:05:17 Waiting for src bucket item count to get to: 200000 curr count: 143057
2021/12/02 17:05:22 Waiting for src bucket item count to get to: 200000 curr count: 159092
2021/12/02 17:05:28 Waiting for src bucket item count to get to: 200000 curr count: 159092
2021/12/02 17:05:33 Waiting for src bucket item count to get to: 200000 curr count: 173745
2021/12/02 17:05:38 Waiting for src bucket item count to get to: 200000 curr count: 173745
2021/12/02 17:05:43 Waiting for src bucket item count to get to: 200000 curr count: 189215
2021/12/02 17:05:48 Waiting for src bucket item count to get to: 200000 curr count: 189215
2021/12/02 17:05:53 Waiting for src bucket item count to get to: 200000 curr count: 189215
2021/12/02 17:05:58 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 17:05:58 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 17:05:58 Starting bucket ops to source bucket
2021/12/02 17:09:50 Waiting for dst bucket item count to get to: 0 curr count: 10583
2021/12/02 17:09:55 Waiting for dst bucket item count to get to: 0 curr count: 10583
2021/12/02 17:09:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:09:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 17:09:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 17:10:00 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 17:10:05 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 17:10:10 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 17:10:15 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 17:10:15 App: TestEventingSwapRebWithOps got un-deployed
2021/12/02 17:10:15 No eventing-con process running
2021/12/02 17:10:15 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/02 17:10:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:10:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:10:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:10:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:10:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:10:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (462.51s)
=== RUN   TestKVRebInWithOps
2021/12/02 17:10:39 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 17:10:39 App: TestKVRebInWithOps got un-deployed
2021/12/02 17:10:39 No eventing-con process running
2021/12/02 17:10:39 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/02 17:10:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:10:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:10:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:10:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:10:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:10:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:10:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:10:57 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 17:10:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 17:11:04 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 17:11:09 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 17:11:14 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 17:11:19 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 17:11:19 App: TestKVRebInWithOps got deployed
2021/12/02 17:11:19 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 17:11:19 Starting bucket ops to source bucket
2021/12/02 17:11:19 Removing node(s): [] from the cluster
2021/12/02 17:11:24 Rebalance progress: 12.5
2021/12/02 17:11:29 Rebalance progress: 12.5
2021/12/02 17:11:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:11:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:11:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:11:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 17:11:39 Rebalance progress: 12.5
2021/12/02 17:11:44 Rebalance progress: 12.5
2021/12/02 17:11:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:11:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:11:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:11:49 Waiting for src bucket item count to get to: 200000 curr count: 30545
2021/12/02 17:11:54 Waiting for src bucket item count to get to: 200000 curr count: 46329
2021/12/02 17:11:59 Waiting for src bucket item count to get to: 200000 curr count: 46329
2021/12/02 17:12:04 Waiting for src bucket item count to get to: 200000 curr count: 61826
2021/12/02 17:12:09 Waiting for src bucket item count to get to: 200000 curr count: 61826
2021/12/02 17:12:14 Waiting for src bucket item count to get to: 200000 curr count: 76684
2021/12/02 17:12:19 Waiting for src bucket item count to get to: 200000 curr count: 76684
2021/12/02 17:12:24 Waiting for src bucket item count to get to: 200000 curr count: 92355
2021/12/02 17:12:29 Waiting for src bucket item count to get to: 200000 curr count: 92355
2021/12/02 17:12:34 Waiting for src bucket item count to get to: 200000 curr count: 108291
2021/12/02 17:12:39 Waiting for src bucket item count to get to: 200000 curr count: 108291
2021/12/02 17:12:44 Waiting for src bucket item count to get to: 200000 curr count: 108291
2021/12/02 17:12:49 Waiting for src bucket item count to get to: 200000 curr count: 124407
2021/12/02 17:12:54 Waiting for src bucket item count to get to: 200000 curr count: 124407
2021/12/02 17:12:59 Waiting for src bucket item count to get to: 200000 curr count: 140563
2021/12/02 17:13:04 Waiting for src bucket item count to get to: 200000 curr count: 156710
2021/12/02 17:13:09 Waiting for src bucket item count to get to: 200000 curr count: 156710
2021/12/02 17:13:14 Waiting for src bucket item count to get to: 200000 curr count: 171527
2021/12/02 17:13:19 Waiting for src bucket item count to get to: 200000 curr count: 171527
2021/12/02 17:13:24 Waiting for src bucket item count to get to: 200000 curr count: 187391
2021/12/02 17:13:29 Waiting for src bucket item count to get to: 200000 curr count: 187391
2021/12/02 17:13:34 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 17:13:34 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 17:13:34 Starting bucket ops to source bucket
2021/12/02 17:17:27 Waiting for dst bucket item count to get to: 0 curr count: 22668
2021/12/02 17:17:32 Waiting for dst bucket item count to get to: 0 curr count: 4700
2021/12/02 17:17:37 Waiting for dst bucket item count to get to: 0 curr count: 4700
2021/12/02 17:17:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:17:37 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 17:17:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 17:17:42 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 17:17:47 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 17:17:52 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 17:17:57 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 17:17:57 App: TestKVRebInWithOps got un-deployed
2021/12/02 17:17:57 No eventing-con process running
2021/12/02 17:17:57 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/02 17:17:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:18:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:18:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:18:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:18:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:18:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (461.83s)
=== RUN   TestKVRebOutWithOps
2021/12/02 17:18:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 17:18:16 Removing node(s): [] from the cluster
2021/12/02 17:18:21 Rebalance progress: 100
2021/12/02 17:18:26 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 17:18:26 App: TestKVRebOutWithOps got un-deployed
2021/12/02 17:18:26 No eventing-con process running
2021/12/02 17:18:26 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/02 17:18:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:18:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:18:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:18:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:18:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:18:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:18:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:18:45 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 17:18:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 17:18:51 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 17:18:56 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 17:19:01 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 17:19:06 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 17:19:06 App: TestKVRebOutWithOps got deployed
2021/12/02 17:19:06 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 17:19:06 Starting bucket ops to source bucket
2021/12/02 17:19:11 Rebalance progress: 12.5
2021/12/02 17:19:16 Rebalance progress: 12.5
2021/12/02 17:19:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:19:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:19:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:19:21 Waiting for src bucket item count to get to: 200000 curr count: 20430
2021/12/02 17:19:26 Waiting for src bucket item count to get to: 200000 curr count: 20430
2021/12/02 17:19:31 Waiting for src bucket item count to get to: 200000 curr count: 36230
2021/12/02 17:19:36 Waiting for src bucket item count to get to: 200000 curr count: 36230
2021/12/02 17:19:41 Waiting for src bucket item count to get to: 200000 curr count: 52515
2021/12/02 17:19:46 Waiting for src bucket item count to get to: 200000 curr count: 52515
2021/12/02 17:19:51 Waiting for src bucket item count to get to: 200000 curr count: 52515
2021/12/02 17:19:56 Waiting for src bucket item count to get to: 200000 curr count: 68585
2021/12/02 17:20:01 Waiting for src bucket item count to get to: 200000 curr count: 68585
2021/12/02 17:20:06 Waiting for src bucket item count to get to: 200000 curr count: 83525
2021/12/02 17:20:11 Waiting for src bucket item count to get to: 200000 curr count: 83525
2021/12/02 17:20:16 Waiting for src bucket item count to get to: 200000 curr count: 99131
2021/12/02 17:20:21 Waiting for src bucket item count to get to: 200000 curr count: 99131
2021/12/02 17:20:26 Waiting for src bucket item count to get to: 200000 curr count: 114834
2021/12/02 17:20:31 Waiting for src bucket item count to get to: 200000 curr count: 114834
2021/12/02 17:20:36 Waiting for src bucket item count to get to: 200000 curr count: 130747
2021/12/02 17:20:41 Waiting for src bucket item count to get to: 200000 curr count: 130747
2021/12/02 17:20:46 Waiting for src bucket item count to get to: 200000 curr count: 146715
2021/12/02 17:20:51 Waiting for src bucket item count to get to: 200000 curr count: 146715
2021/12/02 17:20:56 Waiting for src bucket item count to get to: 200000 curr count: 162865
2021/12/02 17:21:01 Waiting for src bucket item count to get to: 200000 curr count: 162865
2021/12/02 17:21:06 Waiting for src bucket item count to get to: 200000 curr count: 178105
2021/12/02 17:21:11 Waiting for src bucket item count to get to: 200000 curr count: 178105
2021/12/02 17:21:16 Waiting for src bucket item count to get to: 200000 curr count: 193815
2021/12/02 17:21:21 Waiting for src bucket item count to get to: 200000 curr count: 193815
2021/12/02 17:21:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 17:21:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 17:21:26 Starting bucket ops to source bucket
2021/12/02 17:25:17 Waiting for dst bucket item count to get to: 0 curr count: 5754
2021/12/02 17:25:22 Waiting for dst bucket item count to get to: 0 curr count: 5754
2021/12/02 17:25:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:25:22 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 17:25:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 17:25:28 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 17:25:33 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 17:25:38 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 17:25:43 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 17:25:43 App: TestKVRebOutWithOps got un-deployed
2021/12/02 17:25:43 No eventing-con process running
2021/12/02 17:25:43 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/02 17:25:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:25:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:25:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:25:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:25:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:25:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (465.59s)
=== RUN   TestKVSwapRebWithOps
2021/12/02 17:26:06 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 17:26:06 App: TestKVSwapRebWithOps got un-deployed
2021/12/02 17:26:06 No eventing-con process running
2021/12/02 17:26:06 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/02 17:26:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:26:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:26:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:26:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:26:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:26:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:26:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:26:25 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 17:26:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 17:26:31 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 17:26:36 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 17:26:41 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 17:26:46 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 17:26:46 App: TestKVSwapRebWithOps got deployed
2021/12/02 17:26:46 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 17:26:46 Starting bucket ops to source bucket
2021/12/02 17:26:46 Removing node(s): [] from the cluster
2021/12/02 17:26:51 Rebalance progress: 12.5
2021/12/02 17:26:56 Rebalance progress: 12.5
2021/12/02 17:27:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:27:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:27:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:27:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 17:27:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 17:27:07 Rebalance progress: 12.5
2021/12/02 17:27:12 Rebalance progress: 12.5
2021/12/02 17:27:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:27:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:27:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:27:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 17:27:22 Rebalance progress: 12.5
2021/12/02 17:27:27 Rebalance progress: 12.5
2021/12/02 17:27:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:27:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:27:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:27:32 Waiting for src bucket item count to get to: 200000 curr count: 65841
2021/12/02 17:27:37 Waiting for src bucket item count to get to: 200000 curr count: 65841
2021/12/02 17:27:42 Waiting for src bucket item count to get to: 200000 curr count: 80594
2021/12/02 17:27:47 Waiting for src bucket item count to get to: 200000 curr count: 80594
2021/12/02 17:27:52 Waiting for src bucket item count to get to: 200000 curr count: 96525
2021/12/02 17:27:57 Waiting for src bucket item count to get to: 200000 curr count: 96525
2021/12/02 17:28:02 Waiting for src bucket item count to get to: 200000 curr count: 112407
2021/12/02 17:28:07 Waiting for src bucket item count to get to: 200000 curr count: 112407
2021/12/02 17:28:12 Waiting for src bucket item count to get to: 200000 curr count: 112407
2021/12/02 17:28:17 Waiting for src bucket item count to get to: 200000 curr count: 128394
2021/12/02 17:28:22 Waiting for src bucket item count to get to: 200000 curr count: 128394
2021/12/02 17:28:27 Waiting for src bucket item count to get to: 200000 curr count: 144589
2021/12/02 17:28:32 Waiting for src bucket item count to get to: 200000 curr count: 144589
2021/12/02 17:28:37 Waiting for src bucket item count to get to: 200000 curr count: 160821
2021/12/02 17:28:42 Waiting for src bucket item count to get to: 200000 curr count: 160821
2021/12/02 17:28:47 Waiting for src bucket item count to get to: 200000 curr count: 175871
2021/12/02 17:28:52 Waiting for src bucket item count to get to: 200000 curr count: 175871
2021/12/02 17:28:57 Waiting for src bucket item count to get to: 200000 curr count: 191473
2021/12/02 17:29:02 Waiting for src bucket item count to get to: 200000 curr count: 191473
2021/12/02 17:29:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 17:29:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 17:29:07 Starting bucket ops to source bucket
2021/12/02 17:33:00 Waiting for dst bucket item count to get to: 0 curr count: 10099
2021/12/02 17:33:05 Waiting for dst bucket item count to get to: 0 curr count: 10099
2021/12/02 17:33:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:33:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 17:33:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 17:33:10 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 17:33:15 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 17:33:20 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 17:33:25 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 17:33:25 App: TestKVSwapRebWithOps got un-deployed
2021/12/02 17:33:25 No eventing-con process running
2021/12/02 17:33:25 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/02 17:33:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:33:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:33:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:33:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:33:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:33:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (462.51s)
=== RUN   TestEventingRebInWithTimers
2021/12/02 17:33:49 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 17:33:49 App: TestEventingRebInWithTimers got un-deployed
2021/12/02 17:33:49 No eventing-con process running
2021/12/02 17:33:49 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/02 17:33:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:33:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:33:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:33:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:34:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:34:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:34:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:34:07 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 17:34:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 17:34:14 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 17:34:19 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 17:34:24 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 17:34:29 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 17:34:29 App: TestEventingRebInWithTimers got deployed
2021/12/02 17:34:29 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 17:34:29 Starting bucket ops to source bucket
2021/12/02 17:34:29 Removing node(s): [] from the cluster
2021/12/02 17:34:34 Rebalance progress: 12.5
2021/12/02 17:34:39 Rebalance progress: 12.5
2021/12/02 17:34:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:34:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:34:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:34:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 17:34:49 Rebalance progress: 12.5
2021/12/02 17:34:54 Rebalance progress: 12.5
2021/12/02 17:34:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:34:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:34:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:34:59 Waiting for src bucket item count to get to: 50000 curr count: 27245
2021/12/02 17:35:04 Waiting for src bucket item count to get to: 50000 curr count: 36298
2021/12/02 17:35:09 Waiting for src bucket item count to get to: 50000 curr count: 36298
2021/12/02 17:35:14 Waiting for src bucket item count to get to: 50000 curr count: 48875
2021/12/02 17:35:19 Waiting for src bucket item count to get to: 50000 curr count: 48875
2021/12/02 17:35:29 Waiting for dst bucket item count to get to: 50000 curr count: 42967
2021/12/02 17:35:34 Waiting for dst bucket item count to get to: 50000 curr count: 42967
2021/12/02 17:35:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 17:35:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 17:35:34 Starting bucket ops to source bucket
2021/12/02 17:36:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:36:50 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:36:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:37:00 Waiting for dst bucket item count to get to: 0 curr count: 44832
2021/12/02 17:37:05 Waiting for dst bucket item count to get to: 0 curr count: 44832
2021/12/02 17:37:10 Waiting for dst bucket item count to get to: 0 curr count: 31344
2021/12/02 17:37:15 Waiting for dst bucket item count to get to: 0 curr count: 31344
2021/12/02 17:37:20 Waiting for dst bucket item count to get to: 0 curr count: 19813
2021/12/02 17:37:25 Waiting for dst bucket item count to get to: 0 curr count: 19813
2021/12/02 17:37:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:37:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 17:37:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 17:37:30 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 17:37:35 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 17:37:40 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 17:37:45 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 17:37:45 App: TestEventingRebInWithTimers got un-deployed
2021/12/02 17:37:45 No eventing-con process running
2021/12/02 17:37:45 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/02 17:37:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:37:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:37:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:37:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:37:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:37:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (259.73s)
=== RUN   TestEventingRebOutWithTimers
2021/12/02 17:38:03 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 17:38:04 Removing node(s): [] from the cluster
2021/12/02 17:38:09 Rebalance progress: 100
2021/12/02 17:38:14 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 17:38:14 App: TestEventingRebOutWithTimers got un-deployed
2021/12/02 17:38:14 No eventing-con process running
2021/12/02 17:38:14 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/02 17:38:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:38:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:38:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:38:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:38:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:38:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:38:32 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 17:38:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 17:38:39 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 17:38:44 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 17:38:49 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 17:38:54 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 17:38:54 App: TestEventingRebOutWithTimers got deployed
2021/12/02 17:38:54 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 17:38:54 Starting bucket ops to source bucket
2021/12/02 17:38:59 Rebalance progress: 12.5
2021/12/02 17:39:04 Rebalance progress: 12.5
2021/12/02 17:39:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:39:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:39:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:39:09 Waiting for src bucket item count to get to: 50000 curr count: 7283
2021/12/02 17:39:14 Waiting for src bucket item count to get to: 50000 curr count: 20711
2021/12/02 17:39:19 Waiting for src bucket item count to get to: 50000 curr count: 20711
2021/12/02 17:39:24 Waiting for src bucket item count to get to: 50000 curr count: 32398
2021/12/02 17:39:29 Waiting for src bucket item count to get to: 50000 curr count: 32398
2021/12/02 17:39:34 Waiting for src bucket item count to get to: 50000 curr count: 43353
2021/12/02 17:39:39 Waiting for src bucket item count to get to: 50000 curr count: 43353
2021/12/02 17:39:44 Waiting for src bucket item count to get to: 50000 curr count: 43353
2021/12/02 17:39:54 Waiting for dst bucket item count to get to: 50000 curr count: 37350
2021/12/02 17:39:59 Waiting for dst bucket item count to get to: 50000 curr count: 37350
2021/12/02 17:40:04 Waiting for dst bucket item count to get to: 50000 curr count: 47247
2021/12/02 17:40:09 Waiting for dst bucket item count to get to: 50000 curr count: 47247
2021/12/02 17:40:09 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 17:40:09 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 17:40:09 Starting bucket ops to source bucket
2021/12/02 17:41:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:41:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:41:30 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:41:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:41:40 Waiting for dst bucket item count to get to: 0 curr count: 36690
2021/12/02 17:41:45 Waiting for dst bucket item count to get to: 0 curr count: 36690
2021/12/02 17:41:50 Waiting for dst bucket item count to get to: 0 curr count: 25315
2021/12/02 17:41:55 Waiting for dst bucket item count to get to: 0 curr count: 25315
2021/12/02 17:42:00 Waiting for dst bucket item count to get to: 0 curr count: 6390
2021/12/02 17:42:05 Waiting for dst bucket item count to get to: 0 curr count: 6390
2021/12/02 17:42:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:42:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 17:42:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 17:42:10 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 17:42:15 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 17:42:20 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 17:42:25 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 17:42:25 App: TestEventingRebOutWithTimers got un-deployed
2021/12/02 17:42:25 No eventing-con process running
2021/12/02 17:42:25 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/02 17:42:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:42:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:42:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:42:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:42:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:42:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (279.87s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/02 17:42:48 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 17:42:48 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/02 17:42:48 No eventing-con process running
2021/12/02 17:42:48 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/02 17:42:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:42:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:42:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:42:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:43:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:43:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:43:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:43:07 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 17:43:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 17:43:13 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 17:43:18 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 17:43:23 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 17:43:28 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 17:43:28 App: TestEventingSwapRebWithTimers got deployed
2021/12/02 17:43:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 17:43:28 Starting bucket ops to source bucket
2021/12/02 17:43:29 Removing node(s): [] from the cluster
2021/12/02 17:43:34 Rebalance progress: 12.5
2021/12/02 17:43:39 Rebalance progress: 12.5
2021/12/02 17:43:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:43:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:43:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:43:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 17:43:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 17:43:50 Rebalance progress: 12.5
2021/12/02 17:43:55 Rebalance progress: 12.5
2021/12/02 17:44:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:44:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:44:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:44:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 17:44:05 Rebalance progress: 12.5
2021/12/02 17:44:10 Rebalance progress: 12.5
2021/12/02 17:44:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:44:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:44:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:44:15 Waiting for src bucket item count to get to: 50000 curr count: 48197
2021/12/02 17:44:20 Waiting for src bucket item count to get to: 50000 curr count: 48197
2021/12/02 17:44:30 Waiting for dst bucket item count to get to: 50000 curr count: 40327
2021/12/02 17:44:35 Waiting for dst bucket item count to get to: 50000 curr count: 40327
2021/12/02 17:44:40 Waiting for dst bucket item count to get to: 50000 curr count: 48394
2021/12/02 17:44:45 Waiting for dst bucket item count to get to: 50000 curr count: 48394
2021/12/02 17:44:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 17:44:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 17:44:45 Starting bucket ops to source bucket
2021/12/02 17:45:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:46:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:46:06 Waiting for dst bucket item count to get to: 0 curr count: 47977
2021/12/02 17:46:11 Waiting for dst bucket item count to get to: 0 curr count: 47977
2021/12/02 17:46:16 Waiting for dst bucket item count to get to: 0 curr count: 34289
2021/12/02 17:46:21 Waiting for dst bucket item count to get to: 0 curr count: 34289
2021/12/02 17:46:26 Waiting for dst bucket item count to get to: 0 curr count: 17037
2021/12/02 17:46:31 Waiting for dst bucket item count to get to: 0 curr count: 17037
2021/12/02 17:46:36 Waiting for dst bucket item count to get to: 0 curr count: 5223
2021/12/02 17:46:41 Waiting for dst bucket item count to get to: 0 curr count: 5223
2021/12/02 17:46:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:46:41 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 17:46:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 17:46:46 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 17:46:51 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 17:46:56 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 17:47:01 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 17:47:01 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/02 17:47:01 No eventing-con process running
2021/12/02 17:47:01 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/02 17:47:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:47:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:47:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:47:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:47:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:47:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (276.12s)
=== RUN   TestKVRebInWithTimers
2021/12/02 17:47:24 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 17:47:25 App: TestKVRebInWithTimers got un-deployed
2021/12/02 17:47:25 No eventing-con process running
2021/12/02 17:47:25 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/02 17:47:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:47:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:47:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:47:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:47:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:47:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:47:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:47:43 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 17:47:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 17:47:49 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 17:47:54 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 17:47:59 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 17:48:04 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 17:48:04 App: TestKVRebInWithTimers got deployed
2021/12/02 17:48:04 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 17:48:04 Starting bucket ops to source bucket
2021/12/02 17:48:05 Removing node(s): [] from the cluster
2021/12/02 17:48:10 Rebalance progress: 12.5
2021/12/02 17:48:15 Rebalance progress: 12.5
2021/12/02 17:48:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:48:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:48:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:48:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 17:48:25 Rebalance progress: 12.5
2021/12/02 17:48:30 Rebalance progress: 12.5
2021/12/02 17:48:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:48:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:48:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:48:35 Waiting for src bucket item count to get to: 50000 curr count: 29458
2021/12/02 17:48:40 Waiting for src bucket item count to get to: 50000 curr count: 29458
2021/12/02 17:48:45 Waiting for src bucket item count to get to: 50000 curr count: 41223
2021/12/02 17:48:50 Waiting for src bucket item count to get to: 50000 curr count: 41223
2021/12/02 17:49:00 Waiting for dst bucket item count to get to: 50000 curr count: 39145
2021/12/02 17:49:05 Waiting for dst bucket item count to get to: 50000 curr count: 39145
2021/12/02 17:49:10 Waiting for dst bucket item count to get to: 50000 curr count: 47105
2021/12/02 17:49:10 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 17:49:10 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 17:49:10 Starting bucket ops to source bucket
2021/12/02 17:50:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:50:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:50:31 Waiting for dst bucket item count to get to: 0 curr count: 49745
2021/12/02 17:50:36 Waiting for dst bucket item count to get to: 0 curr count: 49745
2021/12/02 17:50:41 Waiting for dst bucket item count to get to: 0 curr count: 42453
2021/12/02 17:50:46 Waiting for dst bucket item count to get to: 0 curr count: 42453
2021/12/02 17:50:51 Waiting for dst bucket item count to get to: 0 curr count: 20428
2021/12/02 17:50:56 Waiting for dst bucket item count to get to: 0 curr count: 20428
2021/12/02 17:51:01 Waiting for dst bucket item count to get to: 0 curr count: 3946
2021/12/02 17:51:06 Waiting for dst bucket item count to get to: 0 curr count: 3946
2021/12/02 17:51:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:51:06 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 17:51:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 17:51:11 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 17:51:16 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 17:51:21 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 17:51:26 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 17:51:26 App: TestKVRebInWithTimers got un-deployed
2021/12/02 17:51:26 No eventing-con process running
2021/12/02 17:51:26 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/02 17:51:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:51:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:51:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:51:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:51:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:51:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (265.28s)
=== RUN   TestKVRebOutWithTimers
2021/12/02 17:51:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 17:51:45 Removing node(s): [] from the cluster
2021/12/02 17:51:50 Rebalance progress: 100
2021/12/02 17:51:55 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 17:51:55 App: TestKVRebOutWithTimers got un-deployed
2021/12/02 17:51:55 No eventing-con process running
2021/12/02 17:51:55 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/02 17:51:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:52:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:52:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:52:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:52:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:52:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:52:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:52:14 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 17:52:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 17:52:20 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 17:52:25 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 17:52:30 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 17:52:35 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 17:52:35 App: TestKVRebOutWithTimers got deployed
2021/12/02 17:52:35 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 17:52:35 Starting bucket ops to source bucket
2021/12/02 17:52:40 Rebalance progress: 12.5
2021/12/02 17:52:45 Rebalance progress: 12.5
2021/12/02 17:52:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:52:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:52:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:52:50 Waiting for src bucket item count to get to: 50000 curr count: 19305
2021/12/02 17:52:55 Waiting for src bucket item count to get to: 50000 curr count: 19305
2021/12/02 17:53:00 Waiting for src bucket item count to get to: 50000 curr count: 30472
2021/12/02 17:53:05 Waiting for src bucket item count to get to: 50000 curr count: 30472
2021/12/02 17:53:10 Waiting for src bucket item count to get to: 50000 curr count: 42523
2021/12/02 17:53:15 Waiting for src bucket item count to get to: 50000 curr count: 42523
2021/12/02 17:53:25 Waiting for dst bucket item count to get to: 50000 curr count: 37414
2021/12/02 17:53:30 Waiting for dst bucket item count to get to: 50000 curr count: 37414
2021/12/02 17:53:35 Waiting for dst bucket item count to get to: 50000 curr count: 37414
2021/12/02 17:53:40 Waiting for dst bucket item count to get to: 50000 curr count: 44955
2021/12/02 17:53:45 Waiting for dst bucket item count to get to: 50000 curr count: 44955
2021/12/02 17:53:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 17:53:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 17:53:45 Starting bucket ops to source bucket
2021/12/02 17:54:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:55:01 Waiting for dst bucket item count to get to: 0 curr count: 49933
2021/12/02 17:55:06 Waiting for dst bucket item count to get to: 0 curr count: 49933
2021/12/02 17:55:11 Waiting for dst bucket item count to get to: 0 curr count: 46816
2021/12/02 17:55:16 Waiting for dst bucket item count to get to: 0 curr count: 46816
2021/12/02 17:55:21 Waiting for dst bucket item count to get to: 0 curr count: 33680
2021/12/02 17:55:26 Waiting for dst bucket item count to get to: 0 curr count: 33680
2021/12/02 17:55:31 Waiting for dst bucket item count to get to: 0 curr count: 15653
2021/12/02 17:55:36 Waiting for dst bucket item count to get to: 0 curr count: 15653
2021/12/02 17:55:41 Waiting for dst bucket item count to get to: 0 curr count: 5094
2021/12/02 17:55:46 Waiting for dst bucket item count to get to: 0 curr count: 5094
2021/12/02 17:55:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:55:46 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 17:55:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 17:55:51 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 17:55:56 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 17:56:01 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 17:56:06 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 17:56:06 App: TestKVRebOutWithTimers got un-deployed
2021/12/02 17:56:06 No eventing-con process running
2021/12/02 17:56:06 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/02 17:56:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:56:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:56:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:56:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:56:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:56:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (280.11s)
=== RUN   TestKVSwapRebWithTimers
2021/12/02 17:56:30 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 17:56:30 App: TestKVSwapRebWithTimers got un-deployed
2021/12/02 17:56:30 No eventing-con process running
2021/12/02 17:56:30 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/02 17:56:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:56:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:56:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 17:56:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:56:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:56:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:56:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 17:56:49 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 17:56:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 17:56:55 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 17:57:00 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 17:57:05 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 17:57:10 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 17:57:10 App: TestKVSwapRebWithTimers got deployed
2021/12/02 17:57:10 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 17:57:10 Starting bucket ops to source bucket
2021/12/02 17:57:10 Removing node(s): [] from the cluster
2021/12/02 17:57:15 Rebalance progress: 12.5
2021/12/02 17:57:20 Rebalance progress: 12.5
2021/12/02 17:57:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:57:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:57:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:57:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 17:57:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 17:57:31 Rebalance progress: 12.5
2021/12/02 17:57:36 Rebalance progress: 12.5
2021/12/02 17:57:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:57:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:57:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:57:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 17:57:46 Rebalance progress: 12.5
2021/12/02 17:57:51 Rebalance progress: 12.5
2021/12/02 17:57:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:57:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 17:57:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 17:57:56 Waiting for src bucket item count to get to: 50000 curr count: 44251
2021/12/02 17:58:06 Waiting for dst bucket item count to get to: 50000 curr count: 42868
2021/12/02 17:58:11 Waiting for dst bucket item count to get to: 50000 curr count: 42868
2021/12/02 17:58:11 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 17:58:11 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 17:58:11 Starting bucket ops to source bucket
2021/12/02 17:59:22 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:59:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:59:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 17:59:37 Waiting for dst bucket item count to get to: 0 curr count: 41161
2021/12/02 17:59:42 Waiting for dst bucket item count to get to: 0 curr count: 41161
2021/12/02 17:59:47 Waiting for dst bucket item count to get to: 0 curr count: 18826
2021/12/02 17:59:52 Waiting for dst bucket item count to get to: 0 curr count: 18826
2021/12/02 17:59:57 Waiting for dst bucket item count to get to: 0 curr count: 7198
2021/12/02 18:00:02 Waiting for dst bucket item count to get to: 0 curr count: 7198
2021/12/02 18:00:07 Waiting for dst bucket item count to get to: 0 curr count: 7198
2021/12/02 18:00:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:00:07 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 18:00:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:00:12 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 18:00:17 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 18:00:22 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 18:00:27 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 18:00:32 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 18:00:37 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 18:00:37 App: TestKVSwapRebWithTimers got un-deployed
2021/12/02 18:00:37 No eventing-con process running
2021/12/02 18:00:37 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/02 18:00:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:00:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:00:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:00:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:00:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:00:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (270.85s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/02 18:01:01 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 18:01:01 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/02 18:01:01 No eventing-con process running
2021/12/02 18:01:01 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/02 18:01:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:01:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:01:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:01:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:01:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:01:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:01:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:01:19 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:01:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 18:01:25 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 18:01:30 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 18:01:35 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 18:01:41 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 18:01:41 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/02 18:01:41 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 18:01:41 Starting bucket ops to source bucket
2021/12/02 18:01:41 Removing node(s): [] from the cluster
2021/12/02 18:01:46 Rebalance progress: 12.5
2021/12/02 18:01:51 Rebalance progress: 12.5
2021/12/02 18:01:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:01:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:01:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:01:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:02:01 Rebalance progress: 12.5
2021/12/02 18:02:06 Rebalance progress: 12.5
2021/12/02 18:02:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:02:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:02:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:02:11 Waiting for src bucket item count to get to: 50000 curr count: 37072
2021/12/02 18:02:16 Waiting for src bucket item count to get to: 50000 curr count: 37072
2021/12/02 18:02:21 Waiting for src bucket item count to get to: 50000 curr count: 46301
2021/12/02 18:02:26 Waiting for src bucket item count to get to: 50000 curr count: 46301
2021/12/02 18:02:36 Waiting for dst bucket item count to get to: 50000 curr count: 28591
2021/12/02 18:02:41 Waiting for dst bucket item count to get to: 50000 curr count: 28591
2021/12/02 18:02:46 Waiting for dst bucket item count to get to: 50000 curr count: 42995
2021/12/02 18:02:51 Waiting for dst bucket item count to get to: 50000 curr count: 42995
2021/12/02 18:02:56 Waiting for dst bucket item count to get to: 50000 curr count: 49877
2021/12/02 18:03:01 Waiting for dst bucket item count to get to: 50000 curr count: 49877
2021/12/02 18:03:01 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:03:01 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:03:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:03:06 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 18:03:11 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 18:03:16 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 18:03:22 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 18:03:27 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 18:03:32 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 18:03:32 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/02 18:03:32 No eventing-con process running
2021/12/02 18:03:32 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/02 18:03:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:03:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:03:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:03:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:03:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:03:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:03:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:04:00 Waiting for dst bucket item count to get to: 0 curr count: 17015
2021/12/02 18:04:05 Waiting for dst bucket item count to get to: 0 curr count: 17015
2021/12/02 18:04:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (194.58s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/02 18:04:10 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 18:04:11 Removing node(s): [] from the cluster
2021/12/02 18:04:16 Rebalance progress: 100
2021/12/02 18:04:21 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 18:04:21 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/02 18:04:21 No eventing-con process running
2021/12/02 18:04:21 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/02 18:04:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:04:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:04:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:04:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:04:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:04:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:04:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:04:39 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:04:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 18:04:46 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 18:04:51 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 18:04:56 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 18:05:01 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 18:05:01 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/02 18:05:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:05:01 Starting bucket ops to source bucket
2021/12/02 18:05:06 Rebalance progress: 12.5
2021/12/02 18:05:11 Rebalance progress: 12.5
2021/12/02 18:05:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:05:16 Waiting for src bucket item count to get to: 50000 curr count: 11120
2021/12/02 18:05:21 Waiting for src bucket item count to get to: 50000 curr count: 24945
2021/12/02 18:05:26 Waiting for src bucket item count to get to: 50000 curr count: 24945
2021/12/02 18:05:31 Waiting for src bucket item count to get to: 50000 curr count: 37286
2021/12/02 18:05:36 Waiting for src bucket item count to get to: 50000 curr count: 37286
2021/12/02 18:05:41 Waiting for src bucket item count to get to: 50000 curr count: 46643
2021/12/02 18:05:46 Waiting for src bucket item count to get to: 50000 curr count: 46643
2021/12/02 18:05:56 Waiting for dst bucket item count to get to: 50000 curr count: 33075
2021/12/02 18:06:01 Waiting for dst bucket item count to get to: 50000 curr count: 33075
2021/12/02 18:06:06 Waiting for dst bucket item count to get to: 50000 curr count: 33075
2021/12/02 18:06:11 Waiting for dst bucket item count to get to: 50000 curr count: 44968
2021/12/02 18:06:16 Waiting for dst bucket item count to get to: 50000 curr count: 44968
2021/12/02 18:06:16 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:06:16 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:06:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:06:21 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 18:06:26 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 18:06:31 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 18:06:36 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 18:06:41 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 18:06:46 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 18:06:46 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/02 18:06:46 No eventing-con process running
2021/12/02 18:06:46 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/02 18:06:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:06:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:06:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:06:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:07:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:07:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:07:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (179.48s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/02 18:07:15 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 18:07:15 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/02 18:07:15 No eventing-con process running
2021/12/02 18:07:15 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/02 18:07:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:07:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:07:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:07:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:07:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:07:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:07:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:07:33 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:07:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 18:07:40 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 18:07:45 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 18:07:50 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 18:07:55 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 18:07:55 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/02 18:07:55 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 18:07:55 Starting bucket ops to source bucket
2021/12/02 18:07:55 Removing node(s): [] from the cluster
2021/12/02 18:08:00 Rebalance progress: 12.5
2021/12/02 18:08:05 Rebalance progress: 12.5
2021/12/02 18:08:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:08:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:08:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:08:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 18:08:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:08:16 Rebalance progress: 12.5
2021/12/02 18:08:21 Rebalance progress: 12.5
2021/12/02 18:08:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:08:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:08:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:08:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 18:08:31 Rebalance progress: 12.5
2021/12/02 18:08:36 Rebalance progress: 12.5
2021/12/02 18:08:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:08:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:08:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:08:46 Waiting for dst bucket item count to get to: 50000 curr count: 31555
2021/12/02 18:08:51 Waiting for dst bucket item count to get to: 50000 curr count: 31555
2021/12/02 18:08:56 Waiting for dst bucket item count to get to: 50000 curr count: 37850
2021/12/02 18:09:01 Waiting for dst bucket item count to get to: 50000 curr count: 37850
2021/12/02 18:09:01 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:09:01 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:09:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:09:06 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 18:09:11 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 18:09:16 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 18:09:21 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 18:09:26 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 18:09:31 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 18:09:31 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/02 18:09:31 No eventing-con process running
2021/12/02 18:09:31 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/02 18:09:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:09:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:09:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:09:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:09:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:09:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:09:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:09:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (170.01s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/02 18:10:05 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 18:10:05 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/02 18:10:05 No eventing-con process running
2021/12/02 18:10:05 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/02 18:10:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:10:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:10:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:10:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:10:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:10:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:10:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:10:23 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:10:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 18:10:29 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 18:10:35 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 18:10:40 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 18:10:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 18:10:45 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/02 18:10:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:10:45 Starting bucket ops to source bucket
2021/12/02 18:10:45 Removing node(s): [] from the cluster
2021/12/02 18:10:50 Rebalance progress: 12.5
2021/12/02 18:10:55 Rebalance progress: 12.5
2021/12/02 18:11:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:11:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:11:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:11:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:11:05 Rebalance progress: 12.5
2021/12/02 18:11:10 Rebalance progress: 12.5
2021/12/02 18:11:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:11:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:11:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:11:15 Waiting for src bucket item count to get to: 50000 curr count: 32095
2021/12/02 18:11:20 Waiting for src bucket item count to get to: 50000 curr count: 32095
2021/12/02 18:11:25 Waiting for src bucket item count to get to: 50000 curr count: 42704
2021/12/02 18:11:35 Waiting for dst bucket item count to get to: 50000 curr count: 29537
2021/12/02 18:11:40 Waiting for dst bucket item count to get to: 50000 curr count: 29537
2021/12/02 18:11:45 Waiting for dst bucket item count to get to: 50000 curr count: 35726
2021/12/02 18:11:50 Waiting for dst bucket item count to get to: 50000 curr count: 35726
2021/12/02 18:11:50 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:11:50 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:11:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:11:55 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 18:12:00 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 18:12:05 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 18:12:10 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 18:12:15 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 18:12:20 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 18:12:20 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/02 18:12:20 No eventing-con process running
2021/12/02 18:12:20 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/02 18:12:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:12:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:12:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:12:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:12:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:12:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:12:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:12:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:12:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (174.26s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/02 18:12:54 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:12:54 Removing node(s): [] from the cluster
2021/12/02 18:12:59 Rebalance progress: 100
2021/12/02 18:13:04 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 18:13:04 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/02 18:13:04 No eventing-con process running
2021/12/02 18:13:04 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/02 18:13:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:13:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:13:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:13:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:13:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:13:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:13:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:13:23 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:13:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 18:13:29 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 18:13:34 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 18:13:39 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 18:13:44 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 18:13:44 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/02 18:13:44 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:13:44 Starting bucket ops to source bucket
2021/12/02 18:13:49 Rebalance progress: 12.5
2021/12/02 18:13:54 Rebalance progress: 12.5
2021/12/02 18:13:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:13:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:13:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:13:59 Waiting for src bucket item count to get to: 50000 curr count: 5867
2021/12/02 18:14:04 Waiting for src bucket item count to get to: 50000 curr count: 19767
2021/12/02 18:14:09 Waiting for src bucket item count to get to: 50000 curr count: 19767
2021/12/02 18:14:14 Waiting for src bucket item count to get to: 50000 curr count: 31995
2021/12/02 18:14:19 Waiting for src bucket item count to get to: 50000 curr count: 31995
2021/12/02 18:14:24 Waiting for src bucket item count to get to: 50000 curr count: 42707
2021/12/02 18:14:29 Waiting for src bucket item count to get to: 50000 curr count: 42707
2021/12/02 18:14:34 Waiting for src bucket item count to get to: 50000 curr count: 42707
2021/12/02 18:14:44 Waiting for dst bucket item count to get to: 50000 curr count: 32250
2021/12/02 18:14:49 Waiting for dst bucket item count to get to: 50000 curr count: 32250
2021/12/02 18:14:54 Waiting for dst bucket item count to get to: 50000 curr count: 39245
2021/12/02 18:14:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:14:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:14:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:15:00 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 18:15:05 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 18:15:10 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 18:15:15 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 18:15:20 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 18:15:25 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 18:15:25 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/02 18:15:25 No eventing-con process running
2021/12/02 18:15:25 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/02 18:15:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:15:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:15:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:15:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:15:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:15:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:15:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:15:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (179.34s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/02 18:15:58 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 18:15:58 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/02 18:15:58 No eventing-con process running
2021/12/02 18:15:58 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/02 18:16:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:16:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:16:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:16:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:16:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:16:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:16:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:16:17 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:16:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 18:16:23 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 18:16:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 18:16:33 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 18:16:38 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 18:16:38 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/02 18:16:38 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:16:38 Starting bucket ops to source bucket
2021/12/02 18:16:38 Removing node(s): [] from the cluster
2021/12/02 18:16:43 Rebalance progress: 12.5
2021/12/02 18:16:48 Rebalance progress: 12.5
2021/12/02 18:16:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:16:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:16:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:16:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 18:16:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:16:59 Rebalance progress: 12.5
2021/12/02 18:17:04 Rebalance progress: 12.5
2021/12/02 18:17:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:17:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:17:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:17:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 18:17:14 Rebalance progress: 12.5
2021/12/02 18:17:19 Rebalance progress: 12.5
2021/12/02 18:17:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:17:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:17:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:17:24 Waiting for src bucket item count to get to: 50000 curr count: 46475
2021/12/02 18:17:29 Waiting for src bucket item count to get to: 50000 curr count: 46475
2021/12/02 18:17:39 Waiting for dst bucket item count to get to: 50000 curr count: 38149
2021/12/02 18:17:44 Waiting for dst bucket item count to get to: 50000 curr count: 38149
2021/12/02 18:17:49 Waiting for dst bucket item count to get to: 50000 curr count: 45366
2021/12/02 18:17:54 Waiting for dst bucket item count to get to: 50000 curr count: 45366
2021/12/02 18:17:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:17:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:17:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:17:59 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 18:18:04 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 18:18:09 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 18:18:14 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 18:18:19 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 18:18:24 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 18:18:24 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/02 18:18:24 No eventing-con process running
2021/12/02 18:18:24 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/02 18:18:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:18:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:18:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:18:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:18:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:18:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:18:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:18:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.47s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/02 18:18:58 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 18:18:58 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/02 18:18:58 No eventing-con process running
2021/12/02 18:18:58 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/02 18:19:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:19:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:19:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:19:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:19:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:19:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:19:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:19:17 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:19:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 18:19:23 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 18:19:28 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 18:19:33 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 18:19:38 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 18:19:38 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/02 18:19:38 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 18:19:38 Starting bucket ops to source bucket
2021/12/02 18:19:38 Removing node(s): [] from the cluster
2021/12/02 18:19:43 Rebalance progress: 12.5
2021/12/02 18:19:48 Rebalance progress: 12.5
2021/12/02 18:19:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:19:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:19:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:19:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:19:58 Rebalance progress: 12.5
2021/12/02 18:20:03 Rebalance progress: 12.5
2021/12/02 18:20:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:20:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:20:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:20:08 Waiting for src bucket item count to get to: 50000 curr count: 26775
2021/12/02 18:20:13 Waiting for src bucket item count to get to: 50000 curr count: 36697
2021/12/02 18:20:18 Waiting for src bucket item count to get to: 50000 curr count: 36697
2021/12/02 18:20:23 Waiting for src bucket item count to get to: 50000 curr count: 49339
2021/12/02 18:20:28 Waiting for src bucket item count to get to: 50000 curr count: 49339
2021/12/02 18:20:38 Waiting for dst bucket item count to get to: 50000 curr count: 42465
2021/12/02 18:20:43 Waiting for dst bucket item count to get to: 50000 curr count: 42465
2021/12/02 18:20:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:20:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:20:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:20:48 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 18:20:53 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 18:20:58 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 18:21:03 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 18:21:08 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 18:21:13 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 18:21:13 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/02 18:21:13 No eventing-con process running
2021/12/02 18:21:13 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/02 18:21:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:21:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:21:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:21:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:21:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:21:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:21:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:21:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.19s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/02 18:21:42 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 18:21:42 Removing node(s): [] from the cluster
2021/12/02 18:21:47 Rebalance progress: 100
2021/12/02 18:21:52 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 18:21:52 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/02 18:21:52 No eventing-con process running
2021/12/02 18:21:52 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/02 18:21:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:21:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:21:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:22:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:22:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:22:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:22:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:22:11 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:22:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 18:22:17 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 18:22:22 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 18:22:27 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 18:22:32 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 18:22:32 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/02 18:22:32 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:22:32 Starting bucket ops to source bucket
2021/12/02 18:22:37 Rebalance progress: 12.5
2021/12/02 18:22:42 Rebalance progress: 12.5
2021/12/02 18:22:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:22:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:22:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:22:47 Waiting for src bucket item count to get to: 50000 curr count: 9258
2021/12/02 18:22:52 Waiting for src bucket item count to get to: 50000 curr count: 22247
2021/12/02 18:22:57 Waiting for src bucket item count to get to: 50000 curr count: 22247
2021/12/02 18:23:02 Waiting for src bucket item count to get to: 50000 curr count: 33777
2021/12/02 18:23:07 Waiting for src bucket item count to get to: 50000 curr count: 33777
2021/12/02 18:23:12 Waiting for src bucket item count to get to: 50000 curr count: 44190
2021/12/02 18:23:17 Waiting for src bucket item count to get to: 50000 curr count: 44190
2021/12/02 18:23:22 Waiting for src bucket item count to get to: 50000 curr count: 44190
2021/12/02 18:23:32 Waiting for dst bucket item count to get to: 50000 curr count: 34329
2021/12/02 18:23:37 Waiting for dst bucket item count to get to: 50000 curr count: 34329
2021/12/02 18:23:37 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:23:37 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:23:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:23:42 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 18:23:48 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 18:23:53 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 18:23:58 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 18:24:03 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 18:24:08 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 18:24:08 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/02 18:24:08 No eventing-con process running
2021/12/02 18:24:08 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/02 18:24:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:24:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:24:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:24:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:24:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:24:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:24:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (169.42s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/02 18:24:36 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 18:24:37 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/02 18:24:37 No eventing-con process running
2021/12/02 18:24:37 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/02 18:24:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:24:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:24:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:24:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:24:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:24:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:24:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:24:55 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:24:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 18:25:01 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 18:25:06 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 18:25:11 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 18:25:16 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 18:25:16 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/02 18:25:16 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 18:25:16 Starting bucket ops to source bucket
2021/12/02 18:25:17 Removing node(s): [] from the cluster
2021/12/02 18:25:22 Rebalance progress: 12.5
2021/12/02 18:25:27 Rebalance progress: 12.5
2021/12/02 18:25:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:25:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:25:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:25:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 18:25:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:25:37 Rebalance progress: 12.5
2021/12/02 18:25:42 Rebalance progress: 12.5
2021/12/02 18:25:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:25:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:25:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:25:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 18:25:53 Rebalance progress: 12.5
2021/12/02 18:25:58 Rebalance progress: 12.5
2021/12/02 18:26:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:26:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:26:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:26:03 Waiting for src bucket item count to get to: 50000 curr count: 47443
2021/12/02 18:26:08 Waiting for src bucket item count to get to: 50000 curr count: 47443
2021/12/02 18:26:18 Waiting for dst bucket item count to get to: 50000 curr count: 46081
2021/12/02 18:26:23 Waiting for dst bucket item count to get to: 50000 curr count: 46081
2021/12/02 18:26:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:26:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:26:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:26:28 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 18:26:33 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 18:26:38 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 18:26:43 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 18:26:48 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 18:26:53 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 18:26:53 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/02 18:26:53 No eventing-con process running
2021/12/02 18:26:53 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/02 18:26:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:27:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:27:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:27:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:27:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:27:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:27:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:27:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (170.11s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/02 18:27:27 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 18:27:27 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/02 18:27:27 No eventing-con process running
2021/12/02 18:27:27 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/02 18:27:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:27:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:27:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:27:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:27:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:27:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:27:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:27:45 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:27:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 18:27:51 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 18:27:56 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 18:28:01 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 18:28:06 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 18:28:06 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/02 18:28:06 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:28:06 Starting bucket ops to source bucket
2021/12/02 18:28:07 Removing node(s): [] from the cluster
2021/12/02 18:28:12 Rebalance progress: 12.5
2021/12/02 18:28:17 Rebalance progress: 12.5
2021/12/02 18:28:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:28:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:28:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:28:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:28:27 Rebalance progress: 12.5
2021/12/02 18:28:32 Rebalance progress: 12.5
2021/12/02 18:28:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:28:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:28:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:28:37 Waiting for src bucket item count to get to: 50000 curr count: 26939
2021/12/02 18:28:42 Waiting for src bucket item count to get to: 50000 curr count: 37989
2021/12/02 18:28:47 Waiting for src bucket item count to get to: 50000 curr count: 37989
2021/12/02 18:28:52 Waiting for src bucket item count to get to: 50000 curr count: 49312
2021/12/02 18:28:57 Waiting for src bucket item count to get to: 50000 curr count: 49312
2021/12/02 18:29:07 Waiting for dst bucket item count to get to: 50000 curr count: 44995
2021/12/02 18:29:12 Waiting for dst bucket item count to get to: 50000 curr count: 44995
2021/12/02 18:29:12 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:29:12 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:29:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:29:17 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 18:29:22 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 18:29:27 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 18:29:32 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 18:29:37 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 18:29:42 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 18:29:42 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/02 18:29:42 No eventing-con process running
2021/12/02 18:29:42 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/02 18:29:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:29:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:29:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:29:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:29:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:30:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:30:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:30:11 Waiting for dst bucket item count to get to: 0 curr count: 48007
2021/12/02 18:30:16 Waiting for dst bucket item count to get to: 0 curr count: 48007
2021/12/02 18:30:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (179.40s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/02 18:30:21 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:30:21 Removing node(s): [] from the cluster
2021/12/02 18:30:26 Rebalance progress: 100
2021/12/02 18:30:31 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 18:30:31 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/02 18:30:31 No eventing-con process running
2021/12/02 18:30:31 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/02 18:30:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:30:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:30:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:30:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:30:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:30:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:30:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:30:50 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:30:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 18:30:56 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 18:31:01 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 18:31:06 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 18:31:11 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 18:31:11 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/02 18:31:11 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:31:11 Starting bucket ops to source bucket
2021/12/02 18:31:16 Rebalance progress: 12.5
2021/12/02 18:31:21 Rebalance progress: 12.5
2021/12/02 18:31:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:31:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:31:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:31:26 Waiting for src bucket item count to get to: 50000 curr count: 10189
2021/12/02 18:31:31 Waiting for src bucket item count to get to: 50000 curr count: 23950
2021/12/02 18:31:36 Waiting for src bucket item count to get to: 50000 curr count: 23950
2021/12/02 18:31:41 Waiting for src bucket item count to get to: 50000 curr count: 35027
2021/12/02 18:31:46 Waiting for src bucket item count to get to: 50000 curr count: 35027
2021/12/02 18:31:51 Waiting for src bucket item count to get to: 50000 curr count: 45019
2021/12/02 18:31:56 Waiting for src bucket item count to get to: 50000 curr count: 45019
2021/12/02 18:32:01 Waiting for src bucket item count to get to: 50000 curr count: 45019
2021/12/02 18:32:11 Waiting for dst bucket item count to get to: 50000 curr count: 34394
2021/12/02 18:32:16 Waiting for dst bucket item count to get to: 50000 curr count: 34394
2021/12/02 18:32:21 Waiting for dst bucket item count to get to: 50000 curr count: 49524
2021/12/02 18:32:26 Waiting for dst bucket item count to get to: 50000 curr count: 49524
2021/12/02 18:32:26 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:32:26 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:32:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:32:31 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 18:32:37 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 18:32:42 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 18:32:47 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 18:32:52 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 18:32:57 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 18:32:57 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/02 18:32:57 No eventing-con process running
2021/12/02 18:32:57 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/02 18:32:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:33:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:33:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:33:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:33:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:33:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:33:20 Waiting for dst bucket item count to get to: 0 curr count: 21833
2021/12/02 18:33:25 Waiting for dst bucket item count to get to: 0 curr count: 21833
2021/12/02 18:33:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (189.39s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/02 18:33:35 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 18:33:35 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/02 18:33:35 No eventing-con process running
2021/12/02 18:33:35 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/02 18:33:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:33:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:33:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:33:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:33:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:33:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:33:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:33:54 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:33:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 18:34:00 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 18:34:05 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 18:34:10 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 18:34:15 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 18:34:15 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/02 18:34:15 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:34:15 Starting bucket ops to source bucket
2021/12/02 18:34:16 Removing node(s): [] from the cluster
2021/12/02 18:34:21 Rebalance progress: 12.5
2021/12/02 18:34:26 Rebalance progress: 12.5
2021/12/02 18:34:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:34:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:34:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:34:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 18:34:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:34:36 Rebalance progress: 12.5
2021/12/02 18:34:41 Rebalance progress: 12.5
2021/12/02 18:34:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:34:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:34:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:34:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 18:34:51 Rebalance progress: 12.5
2021/12/02 18:34:56 Rebalance progress: 12.5
2021/12/02 18:35:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:35:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:35:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:35:01 Waiting for src bucket item count to get to: 50000 curr count: 48344
2021/12/02 18:35:06 Waiting for src bucket item count to get to: 50000 curr count: 48344
2021/12/02 18:35:16 Waiting for dst bucket item count to get to: 50000 curr count: 46282
2021/12/02 18:35:21 Waiting for dst bucket item count to get to: 50000 curr count: 46282
2021/12/02 18:35:21 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 18:35:21 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 18:35:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:35:27 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 18:35:32 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 18:35:37 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 18:35:42 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 18:35:47 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 18:35:52 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 18:35:52 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/02 18:35:52 No eventing-con process running
2021/12/02 18:35:52 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/02 18:35:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:35:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:36:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:36:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 18:36:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:36:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:36:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:36:20 Waiting for dst bucket item count to get to: 0 curr count: 24144
2021/12/02 18:36:25 Waiting for dst bucket item count to get to: 0 curr count: 24144
2021/12/02 18:36:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (180.06s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/02 18:36:35 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:36:35 App: bucket_op_on_update got un-deployed
2021/12/02 18:36:35 No eventing-con process running
2021/12/02 18:36:35 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 18:36:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:36:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:36:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:36:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:36:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:36:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:36:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:36:54 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:36:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 18:37:00 Starting bucket ops to source bucket
2021/12/02 18:37:05 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:37:10 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:37:15 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:37:15 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:37:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:37:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:37:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:37:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 18:37:21 Rebalance progress: 12.5
2021/12/02 18:37:26 Rebalance progress: 12.5
2021/12/02 18:37:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:37:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:37:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:37:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 18:37:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 18:37:36 Rebalance progress: 12.5
2021/12/02 18:37:41 Rebalance progress: 12.5
2021/12/02 18:37:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:37:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:37:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:37:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 18:37:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 18:37:51 Rebalance progress: 12.5
2021/12/02 18:37:56 Rebalance progress: 12.5
2021/12/02 18:38:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:38:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:38:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:38:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:38:07 Rebalance progress: 12.5
2021/12/02 18:38:12 Rebalance progress: 12.5
2021/12/02 18:38:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:38:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:38:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:38:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 18:38:22 Rebalance progress: 12.5
2021/12/02 18:38:27 Rebalance progress: 12.5
2021/12/02 18:38:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:38:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:38:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:38:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 18:38:37 Rebalance progress: 12.5
2021/12/02 18:38:42 Rebalance progress: 12.5
2021/12/02 18:38:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:38:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:38:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:38:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:38:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:38:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:39:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:39:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:39:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:39:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:39:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:39:17 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:39:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 18:39:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:39:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:39:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10649
2021/12/02 18:39:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:39:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:39:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:39:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8722
2021/12/02 18:39:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8722
2021/12/02 18:39:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.40s)
=== RUN   TestKVForceFailover
2021/12/02 18:40:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:40:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:40:01 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:40:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 18:40:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:40:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:40:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:40:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:40:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:40:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:40:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:40:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 18:40:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:40:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:40:20 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:40:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 18:40:21 Starting bucket ops to source bucket
2021/12/02 18:40:26 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:40:31 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:40:36 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:40:41 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:40:41 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:40:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:40:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:40:41 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/02 18:41:01 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/02 18:41:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 18:41:07 Rebalance progress: 12.5
2021/12/02 18:41:12 Rebalance progress: 12.5
2021/12/02 18:41:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:41:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:41:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:41:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/02 18:41:22 Rebalance progress: 12.5
2021/12/02 18:41:27 Rebalance progress: 12.5
2021/12/02 18:41:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:41:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:41:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:41:32 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:41:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:41:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:41:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:41:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:41:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:42:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:42:03 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:42:03 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:42:03 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 18:42:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:42:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:42:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7079
2021/12/02 18:42:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:42:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:42:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:42:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7079
2021/12/02 18:42:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (160.30s)
=== RUN   TestKVHardFailoverRecovery
2021/12/02 18:42:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:42:36 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:42:36 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:42:36 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 18:42:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:42:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:42:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:42:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:42:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:42:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:42:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:42:55 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:42:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 18:43:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 18:43:06 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:43:11 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:43:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:43:16 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:43:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:43:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:43:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:43:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 18:43:21 Rebalance progress: 12.5
2021/12/02 18:43:26 Rebalance progress: 12.5
2021/12/02 18:43:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:43:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:43:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:43:31 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/02 18:44:31 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/02 18:44:36 Rebalance progress: 12.5
2021/12/02 18:44:41 Rebalance progress: 12.5
2021/12/02 18:44:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:44:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:44:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:44:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:44:52 Rebalance progress: 12.5
2021/12/02 18:44:57 Rebalance progress: 12.5
2021/12/02 18:45:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:45:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:45:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:45:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:45:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:45:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:45:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:45:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:45:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:45:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:45:32 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:45:32 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:45:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 18:45:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:45:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:45:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12039
2021/12/02 18:45:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12039
2021/12/02 18:45:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:45:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:45:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:46:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7470
2021/12/02 18:46:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7470
2021/12/02 18:46:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (219.41s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/02 18:46:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:46:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:46:16 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:46:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 18:46:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:46:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:46:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:46:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:46:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:46:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:46:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:46:34 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:46:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 18:46:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 18:46:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:46:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:46:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:46:55 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:46:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:46:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:46:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:46:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 18:46:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 18:46:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 18:47:02 Rebalance progress: 12.5
2021/12/02 18:47:07 Rebalance progress: 12.5
2021/12/02 18:47:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:47:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:47:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:47:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/02 18:47:17 Rebalance progress: 12.5
2021/12/02 18:47:22 Rebalance progress: 12.5
2021/12/02 18:47:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:47:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:47:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:47:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:47:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:47:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:47:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:47:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:47:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:47:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:47:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:47:57 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:47:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 18:47:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:48:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:48:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4648
2021/12/02 18:48:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:48:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:48:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:48:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4077
2021/12/02 18:48:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4077
2021/12/02 18:48:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (140.34s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/02 18:48:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:48:36 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:48:36 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:48:36 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 18:48:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:48:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:48:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:48:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:48:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:48:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:48:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:48:50 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:48:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 18:48:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 18:49:01 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:49:06 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:49:11 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:49:11 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:49:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:49:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:49:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:49:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 18:49:16 Rebalance progress: 12.5
2021/12/02 18:49:21 Rebalance progress: 12.5
2021/12/02 18:49:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:49:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:49:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:49:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 18:49:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:49:32 Rebalance progress: 12.5
2021/12/02 18:49:37 Rebalance progress: 12.5
2021/12/02 18:49:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:49:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:49:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:49:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 18:49:47 Rebalance progress: 12.5
2021/12/02 18:49:52 Rebalance progress: 12.5
2021/12/02 18:49:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:49:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:49:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:49:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:50:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:50:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:50:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:50:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:50:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:50:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:50:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:50:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:50:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 18:50:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:50:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:50:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6085
2021/12/02 18:50:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:50:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:50:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:50:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4366
2021/12/02 18:50:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4366
2021/12/02 18:50:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:51:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:51:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:51:01 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:51:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 18:51:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:51:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:51:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:51:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:51:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:51:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.35s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/02 18:51:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:51:24 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:51:24 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:51:24 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 18:51:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:51:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:51:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:51:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:51:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:51:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:51:43 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:51:43 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 18:51:43 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:51:43 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/02 18:51:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:51:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:51:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:51:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:51:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:51:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:51:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:51:56 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:51:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:51:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:51:58 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:51:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/02 18:52:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 18:52:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:52:14 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:52:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:52:19 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/02 18:52:24 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 18:52:29 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 18:52:34 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 18:52:34 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:52:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:52:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:52:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:52:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 18:52:39 Rebalance progress: 12.5
2021/12/02 18:52:44 Rebalance progress: 12.5
2021/12/02 18:52:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:52:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:52:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:52:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 18:52:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 18:52:54 Rebalance progress: 12.5
2021/12/02 18:52:59 Rebalance progress: 12.5
2021/12/02 18:53:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:53:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:53:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:53:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 18:53:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 18:53:10 Rebalance progress: 12.5
2021/12/02 18:53:15 Rebalance progress: 12.5
2021/12/02 18:53:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:53:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:53:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:53:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:53:25 Rebalance progress: 12.5
2021/12/02 18:53:30 Rebalance progress: 12.5
2021/12/02 18:53:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:53:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:53:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:53:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 18:53:40 Rebalance progress: 12.5
2021/12/02 18:53:45 Rebalance progress: 12.5
2021/12/02 18:53:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:53:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:53:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:53:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 18:53:55 Rebalance progress: 12.5
2021/12/02 18:54:00 Rebalance progress: 12.5
2021/12/02 18:54:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:54:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:54:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:54:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:54:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:54:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:54:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:54:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:54:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:54:30 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:54:30 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:54:31 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 18:54:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:54:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:54:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12130
2021/12/02 18:54:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12130
2021/12/02 18:54:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:54:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:54:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:55:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7446
2021/12/02 18:55:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7446
2021/12/02 18:55:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:55:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:55:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:55:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:55:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:55:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:55:31 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:55:36 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:55:36 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 18:55:36 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:55:36 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/02 18:55:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:55:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:55:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:55:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:55:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:55:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (275.25s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/02 18:56:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:56:00 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:56:00 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:56:00 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 18:56:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:56:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:56:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:56:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:56:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:56:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:56:18 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:56:18 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 18:56:18 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:56:18 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/02 18:56:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:56:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:56:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:56:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:56:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:56:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:56:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:56:32 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:56:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:56:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:56:33 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 18:56:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/02 18:56:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 18:56:44 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:56:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:56:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 18:56:54 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/02 18:56:59 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 18:57:04 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 18:57:09 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 18:57:09 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:57:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:57:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:57:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 18:57:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 18:57:15 Rebalance progress: 12.5
2021/12/02 18:57:20 Rebalance progress: 12.5
2021/12/02 18:57:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:57:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:57:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:57:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 18:57:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 18:57:30 Rebalance progress: 12.5
2021/12/02 18:57:35 Rebalance progress: 12.5
2021/12/02 18:57:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:57:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:57:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:57:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 18:57:45 Rebalance progress: 12.5
2021/12/02 18:57:50 Rebalance progress: 12.5
2021/12/02 18:57:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:57:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:57:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 18:57:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:58:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:58:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:58:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:58:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:58:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:58:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:58:21 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:58:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 18:58:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:58:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:58:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7615
2021/12/02 18:58:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7615
2021/12/02 18:58:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:58:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:58:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:58:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4478
2021/12/02 18:58:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4478
2021/12/02 18:58:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:58:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 18:59:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:59:06 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:59:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:59:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:59:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:59:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 18:59:26 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 18:59:26 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:59:26 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/02 18:59:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:59:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:59:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:59:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:59:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:59:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (230.08s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/02 18:59:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 18:59:50 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 18:59:50 eventingSetup.go:46: No eventing-con process running
2021/12/02 18:59:50 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 18:59:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 18:59:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 18:59:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 18:59:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:00:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:00:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:00:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:00:08 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 19:00:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 19:00:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 19:00:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 19:00:24 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 19:00:29 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 19:00:29 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:00:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:00:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:00:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 19:00:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 19:00:50 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:00:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:00:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:01:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:01:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:01:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 19:01:06 Rebalance progress: 12.5
2021/12/02 19:01:11 Rebalance progress: 12.5
2021/12/02 19:01:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:01:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:01:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:01:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 19:01:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 19:01:21 Rebalance progress: 12.5
2021/12/02 19:01:26 Rebalance progress: 12.5
2021/12/02 19:01:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:01:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:01:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:01:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 19:01:36 Rebalance progress: 12.5
2021/12/02 19:01:41 Rebalance progress: 12.5
2021/12/02 19:01:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:01:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:01:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:01:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 19:01:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:01:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:02:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:02:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:02:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:02:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:02:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 19:02:17 eventingSetup.go:46: No eventing-con process running
2021/12/02 19:02:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 19:02:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:02:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:02:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9185
2021/12/02 19:02:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 19:02:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:02:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:02:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3533
2021/12/02 19:02:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3533
2021/12/02 19:02:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.70s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/02 19:02:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:02:55 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 19:02:55 eventingSetup.go:46: No eventing-con process running
2021/12/02 19:02:55 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 19:02:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:03:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:03:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 19:03:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:03:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:03:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:03:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:03:14 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 19:03:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 19:03:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 19:03:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 19:03:30 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 19:03:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 19:03:35 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:03:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:03:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:03:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 19:03:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 19:03:40 Rebalance progress: 12.5
2021/12/02 19:03:45 Rebalance progress: 12.5
2021/12/02 19:03:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:03:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:03:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:03:50 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:04:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:04:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:04:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 19:04:06 Rebalance progress: 12.5
2021/12/02 19:04:11 Rebalance progress: 12.5
2021/12/02 19:04:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:04:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:04:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:04:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 19:04:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 19:04:21 Rebalance progress: 12.5
2021/12/02 19:04:26 Rebalance progress: 12.5
2021/12/02 19:04:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:04:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:04:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:04:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 19:04:36 Rebalance progress: 12.5
2021/12/02 19:04:41 Rebalance progress: 12.5
2021/12/02 19:04:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:04:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:04:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:04:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 19:04:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:04:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:05:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:05:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:05:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:05:12 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 19:05:12 eventingSetup.go:46: No eventing-con process running
2021/12/02 19:05:12 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 19:05:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:05:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:05:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8622
2021/12/02 19:05:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8622
2021/12/02 19:05:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 19:05:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:05:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:05:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3464
2021/12/02 19:05:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3464
2021/12/02 19:05:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (180.07s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/02 19:05:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 19:05:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 19:05:56 Rebalance progress: 100
2021/12/02 19:05:56 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:06:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:06:26 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 19:06:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 19:06:32 Removing node(s): [] from the cluster
2021/12/02 19:06:32 Starting bucket ops to source bucket
2021/12/02 19:06:37 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/02 19:06:42 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 19:06:47 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 19:06:47 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:06:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:06:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 19:06:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 19:06:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:06:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:07:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:07:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:07:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:07:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 19:07:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 19:07:17 eventingSetup.go:46: No eventing-con process running
2021/12/02 19:07:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 19:07:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:07:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:07:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1505
2021/12/02 19:07:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 19:07:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 19:07:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 19:07:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1505
2021/12/02 19:07:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1505
2021/12/02 19:07:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.68s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12193.581s

Rebalance tests took 3 hours, 23 minutes and 16 seconds.

Testing Succeeded


Version: versions-02.12.2021-11.23.cfg
Build Log: make-02.12.2021-11.23.log
Server Log: logs-02.12.2021-11.23.tar.gz

Finished