Building

Started building at 2021/12/16 18:14:22
Calculating base
Updating mirror
Basing run on 7.1.0-1926 b9c3e52ad
Updating tree for run 16.12.2021-18.14
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at f0d7a1e, changes since last good build: none
kv_engine is at 0811a49, changes since last good build: none
ns_server is at 009dd9f, changes since last good build: none
couchstore is at 70053f3, changes since last good build: 
 70053f3 MB-49820: Ensure we always decode keys using namespace
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 23b59c2, changes since last good build: 
 23b59c26c Test case automation for GSI autofailover
Switching eventing to unstable
eventing is at 7f52754, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 3f826e0, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 694a5510] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 8185562b] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/18
Total 4 (delta 3), reused 4 (delta 3)
[unstable 2d1835d8] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 58abf998] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 167196 at refs/changes/96/167196/8
Total 19 (delta 15), reused 18 (delta 15)
[unstable 562fda40] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 158 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167542 at refs/changes/42/167542/5
Total 44 (delta 38), reused 44 (delta 38)
[unstable fda83357] MB-49570: Start dcp stream for a single collection id
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:27:25 2021 +0530
 7 files changed, 52 insertions(+), 29 deletions(-)
Pulling in uncommitted change 167544 at refs/changes/44/167544/4
Total 52 (delta 44), reused 52 (delta 44)
[unstable 769c74f3] MB-49570: Add backlog stats on ui
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:28:01 2021 +0530
 2 files changed, 18 insertions(+), 1 deletion(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/16 18:30:10
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/16 18:31:42 Started functional
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:44870->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:44870->127.0.0.1:9000: read: connection reset by peer
2021/12/16 18:31:52 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/16 18:31:58 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/16 18:31:58 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 18:32:03 Removing node(s): [] from the cluster
2021/12/16 18:32:08 Rebalance progress: 0
2021/12/16 18:32:13 Rebalance progress: 1.157407407407407
2021/12/16 18:32:18 Rebalance progress: 5.787037037037037
2021/12/16 18:32:24 Rebalance progress: 9.25925925925926
2021/12/16 18:32:28 Rebalance progress: 13.88888888888889
2021/12/16 18:32:33 Rebalance progress: 18.51851851851852
2021/12/16 18:32:38 Rebalance progress: 21.99074074074074
2021/12/16 18:32:43 Rebalance progress: 26.62037037037037
2021/12/16 18:32:48 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/16 18:32:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:33:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:33:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:33:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:33:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:34:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:34:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:34:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:34:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:35:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:35:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:35:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:35:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:36:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:36:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:36:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:36:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:37:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:37:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:37:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:37:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 18:38:09 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:38:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/16 18:38:15 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 18:38:20 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 18:38:25 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 18:38:31 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 18:38:31 App: TestAdvancedGetOps got deployed
2021/12/16 18:38:31 Starting bucket ops to source bucket
2021/12/16 18:38:36 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 18:38:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 18:38:46 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 18:38:46 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 18:38:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:38:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:38:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:38:51 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 18:38:56 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 18:39:01 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 18:39:06 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 18:39:06 App: TestAdvancedGetOps got un-deployed
2021/12/16 18:39:06 No eventing-con process running
2021/12/16 18:39:06 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/16 18:39:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:39:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 18:39:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:39:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:39:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 18:39:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 18:39:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (400.22s)
=== RUN   TestAdvancedInsertOps
2021/12/16 18:39:34 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:39:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 18:39:41 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 18:39:46 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 18:39:51 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 18:39:56 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 18:39:56 App: TestAdvancedInsertOps got deployed
2021/12/16 18:39:56 Starting bucket ops to source bucket
2021/12/16 18:40:01 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/16 18:40:01 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/16 18:40:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:40:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:40:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:40:07 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 18:40:12 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 18:40:17 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 18:40:22 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 18:40:22 App: TestAdvancedInsertOps got un-deployed
2021/12/16 18:40:22 No eventing-con process running
2021/12/16 18:40:22 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/16 18:40:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:40:33 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 18:40:38 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/16 18:40:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:40:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:40:50 Waiting for dst bucket item count to get to: 0 curr count: 732
2021/12/16 18:40:55 Waiting for dst bucket item count to get to: 0 curr count: 732
2021/12/16 18:40:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:40:55 Testing insert operation on source bucket
2021/12/16 18:40:55 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:40:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 18:41:01 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 18:41:06 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 18:41:11 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 18:41:16 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 18:41:16 App: TestAdvancedInsertOps got deployed
2021/12/16 18:41:16 Starting bucket ops to source bucket
2021/12/16 18:41:22 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/16 18:41:27 Waiting for dst bucket item count to get to: 2048 curr count: 1899
2021/12/16 18:41:32 Waiting for dst bucket item count to get to: 2048 curr count: 1899
2021/12/16 18:41:32 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/16 18:41:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:41:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:41:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:41:37 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 18:41:42 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 18:41:47 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 18:41:52 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 18:41:52 App: TestAdvancedInsertOps got un-deployed
2021/12/16 18:41:52 No eventing-con process running
2021/12/16 18:41:52 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/16 18:41:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:42:04 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/16 18:42:09 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 18:42:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:42:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:42:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (161.55s)
=== RUN   TestAdvancedUpsertOps
2021/12/16 18:42:16 Starting bucket ops to source bucket
2021/12/16 18:42:17 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:42:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 18:42:23 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 18:42:28 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 18:42:33 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 18:42:38 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 18:42:38 App: TestAdvancedUpsertOps got deployed
2021/12/16 18:42:38 Starting bucket ops to source bucket
2021/12/16 18:42:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 18:42:49 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/16 18:42:54 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/16 18:42:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:42:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:42:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:42:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:42:59 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 18:43:04 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 18:43:09 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 18:43:14 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 18:43:14 App: TestAdvancedUpsertOps got un-deployed
2021/12/16 18:43:14 No eventing-con process running
2021/12/16 18:43:14 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/16 18:43:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:43:26 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 18:43:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:43:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:43:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:43:32 Testing upsert operation on source bucket
2021/12/16 18:43:32 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:43:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 18:43:39 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 18:43:44 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 18:43:49 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 18:43:54 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 18:43:54 App: TestAdvancedUpsertOps got deployed
2021/12/16 18:43:54 Starting bucket ops to source bucket
2021/12/16 18:43:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:43:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:43:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:43:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:44:00 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 18:44:05 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 18:44:10 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 18:44:15 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 18:44:15 App: TestAdvancedUpsertOps got un-deployed
2021/12/16 18:44:15 No eventing-con process running
2021/12/16 18:44:15 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/16 18:44:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:44:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:44:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:44:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (133.08s)
=== RUN   TestAdvancedReplaceOps
2021/12/16 18:44:29 Starting bucket ops to source bucket
2021/12/16 18:44:30 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:44:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/16 18:44:36 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 18:44:41 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 18:44:46 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 18:44:51 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 18:44:51 App: TestAdvancedReplaceOps got deployed
2021/12/16 18:44:51 Starting bucket ops to source bucket
2021/12/16 18:44:57 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 18:45:02 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 18:45:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:45:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:45:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:45:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:45:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 18:45:12 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 18:45:17 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 18:45:22 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 18:45:22 App: TestAdvancedReplaceOps got un-deployed
2021/12/16 18:45:22 No eventing-con process running
2021/12/16 18:45:22 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/16 18:45:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:45:34 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 18:45:39 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/16 18:45:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:45:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:45:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:45:46 Testing upsert operation on source bucket
2021/12/16 18:45:46 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:45:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/16 18:45:52 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 18:45:57 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 18:46:02 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 18:46:07 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 18:46:07 App: TestAdvancedReplaceOps got deployed
2021/12/16 18:46:07 Starting bucket ops to source bucket
2021/12/16 18:46:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:46:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:46:08 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:46:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:46:13 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 18:46:18 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 18:46:23 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 18:46:28 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 18:46:28 App: TestAdvancedReplaceOps got un-deployed
2021/12/16 18:46:28 No eventing-con process running
2021/12/16 18:46:29 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/16 18:46:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:46:40 Waiting for dst bucket item count to get to: 0 curr count: 103
2021/12/16 18:46:45 Waiting for dst bucket item count to get to: 0 curr count: 103
2021/12/16 18:46:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:46:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:46:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (142.95s)
=== RUN   TestAdvancedDeleteOps
2021/12/16 18:46:52 Starting bucket ops to source bucket
2021/12/16 18:46:52 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:46:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/16 18:46:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 18:47:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 18:47:08 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 18:47:13 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 18:47:13 App: TestAdvancedDeleteOps got deployed
2021/12/16 18:47:13 Starting bucket ops to source bucket
2021/12/16 18:47:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 18:47:23 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 18:47:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:47:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:47:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:47:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:47:29 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 18:47:34 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 18:47:39 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 18:47:44 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 18:47:44 App: TestAdvancedDeleteOps got un-deployed
2021/12/16 18:47:44 No eventing-con process running
2021/12/16 18:47:44 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/16 18:47:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:47:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 18:47:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:47:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:48:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:48:02 Testing delete operation on source bucket
2021/12/16 18:48:02 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:48:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/16 18:48:08 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 18:48:13 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 18:48:18 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 18:48:23 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 18:48:23 App: TestAdvancedDeleteOps got deployed
2021/12/16 18:48:23 Starting bucket ops to source bucket
2021/12/16 18:48:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:48:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:48:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:48:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:48:29 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 18:48:34 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 18:48:39 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 18:48:44 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 18:48:44 App: TestAdvancedDeleteOps got un-deployed
2021/12/16 18:48:44 No eventing-con process running
2021/12/16 18:48:44 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/16 18:48:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:48:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:48:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:48:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.64s)
=== RUN   TestEnoentAdvancedGet
2021/12/16 18:48:58 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:48:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/16 18:49:04 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 18:49:09 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 18:49:14 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 18:49:19 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 18:49:19 App: TestEnoentAdvancedGet got deployed
2021/12/16 18:49:19 Starting bucket ops to source bucket
2021/12/16 18:49:24 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 18:49:29 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 18:49:29 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 18:49:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:49:29 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:49:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:49:34 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 18:49:39 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 18:49:44 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 18:49:49 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 18:49:49 App: TestEnoentAdvancedGet got un-deployed
2021/12/16 18:49:49 No eventing-con process running
2021/12/16 18:49:49 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/16 18:49:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:50:01 Waiting for dst bucket item count to get to: 0 curr count: 94
2021/12/16 18:50:06 Waiting for dst bucket item count to get to: 0 curr count: 94
2021/12/16 18:50:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:50:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:50:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 18:50:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:50:18 Testing get enoent operation on source bucket
2021/12/16 18:50:18 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:50:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/16 18:50:24 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 18:50:29 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 18:50:34 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 18:50:39 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 18:50:39 App: TestEnoentAdvancedGet got deployed
2021/12/16 18:50:39 Starting bucket ops to source bucket
2021/12/16 18:50:44 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/16 18:50:49 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/16 18:50:49 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/16 18:50:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:50:49 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:50:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:50:54 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 18:50:59 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 18:51:04 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 18:51:09 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 18:51:09 App: TestEnoentAdvancedGet got un-deployed
2021/12/16 18:51:09 No eventing-con process running
2021/12/16 18:51:09 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/16 18:51:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:51:21 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 18:51:26 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 18:51:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:51:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:51:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.08s)
=== RUN   TestEnoentAdvancedReplace
2021/12/16 18:51:33 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:51:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 18:51:39 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 18:51:44 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 18:51:49 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 18:51:54 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 18:51:54 App: TestEnoentAdvancedReplace got deployed
2021/12/16 18:51:54 Starting bucket ops to source bucket
2021/12/16 18:51:59 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 18:52:04 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 18:52:04 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 18:52:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:52:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:52:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:52:09 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 18:52:14 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 18:52:19 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 18:52:24 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 18:52:24 App: TestEnoentAdvancedReplace got un-deployed
2021/12/16 18:52:24 No eventing-con process running
2021/12/16 18:52:24 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/16 18:52:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:52:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 18:52:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:52:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:52:48 Waiting for dst bucket item count to get to: 0 curr count: 52
2021/12/16 18:52:53 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/16 18:52:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:52:53 Testing get enoent operation on source bucket
2021/12/16 18:52:53 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:52:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 18:52:59 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 18:53:04 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 18:53:09 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 18:53:14 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 18:53:14 App: TestEnoentAdvancedReplace got deployed
2021/12/16 18:53:14 Starting bucket ops to source bucket
2021/12/16 18:53:19 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/16 18:53:24 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/16 18:53:24 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/16 18:53:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:53:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:53:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:53:29 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 18:53:34 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 18:53:39 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 18:53:44 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 18:53:44 App: TestEnoentAdvancedReplace got un-deployed
2021/12/16 18:53:44 No eventing-con process running
2021/12/16 18:53:44 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/16 18:53:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:53:56 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 18:53:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:53:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:54:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.00s)
=== RUN   TestEnoentAdvancedDelete
2021/12/16 18:54:03 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:54:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 18:54:09 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 18:54:14 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 18:54:19 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 18:54:24 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 18:54:24 App: TestEnoentAdvancedDelete got deployed
2021/12/16 18:54:24 Starting bucket ops to source bucket
2021/12/16 18:54:29 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 18:54:34 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 18:54:34 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 18:54:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:54:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:54:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:54:39 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 18:54:44 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 18:54:49 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 18:54:54 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 18:54:54 App: TestEnoentAdvancedDelete got un-deployed
2021/12/16 18:54:54 No eventing-con process running
2021/12/16 18:54:54 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/16 18:54:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:55:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 18:55:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:55:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:55:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 18:55:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:55:18 Testing get enoent operation on source bucket
2021/12/16 18:55:18 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:55:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 18:55:24 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 18:55:29 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 18:55:34 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 18:55:39 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 18:55:39 App: TestEnoentAdvancedDelete got deployed
2021/12/16 18:55:39 Starting bucket ops to source bucket
2021/12/16 18:55:44 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/16 18:55:49 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/16 18:55:49 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/16 18:55:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:55:49 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:55:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:55:54 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 18:55:59 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 18:56:04 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 18:56:09 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 18:56:09 App: TestEnoentAdvancedDelete got un-deployed
2021/12/16 18:56:09 No eventing-con process running
2021/12/16 18:56:09 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/16 18:56:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:56:21 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 18:56:26 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 18:56:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:56:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:56:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.05s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/16 18:56:33 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:56:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/16 18:56:39 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 18:56:44 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 18:56:49 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 18:56:54 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 18:56:54 App: TestAdvancedInsertKeyExist got deployed
2021/12/16 18:56:54 Starting bucket ops to source bucket
2021/12/16 18:56:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:56:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:56:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:56:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:56:59 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 18:57:04 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 18:57:09 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 18:57:14 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 18:57:14 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/16 18:57:14 No eventing-con process running
2021/12/16 18:57:14 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/16 18:57:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:57:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:57:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:57:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:57:28 Pumping document in the destination bucket
2021/12/16 18:57:28 Starting bucket ops to source bucket
2021/12/16 18:57:28 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:57:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/16 18:57:35 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 18:57:40 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 18:57:45 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 18:57:50 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 18:57:50 App: TestAdvancedInsertKeyExist got deployed
2021/12/16 18:57:50 Starting bucket ops to source bucket
2021/12/16 18:57:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 18:58:00 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 18:58:05 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 18:58:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:58:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:58:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:58:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:58:10 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 18:58:15 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 18:58:20 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 18:58:25 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 18:58:25 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/16 18:58:25 No eventing-con process running
2021/12/16 18:58:25 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/16 18:58:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:58:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 18:58:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:58:39 Context: Bucket flush, response code: 200 dump: 
2021/12/16 18:58:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.77s)
=== RUN   TestExipryGet
2021/12/16 18:58:44 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:58:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/16 18:58:50 Waiting for app: TestExipryGet to get deployed
2021/12/16 18:58:55 Waiting for app: TestExipryGet to get deployed
2021/12/16 18:59:00 Waiting for app: TestExipryGet to get deployed
2021/12/16 18:59:05 Waiting for app: TestExipryGet to get deployed
2021/12/16 18:59:05 App: TestExipryGet got deployed
2021/12/16 18:59:05 Starting bucket ops to source bucket
2021/12/16 18:59:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 18:59:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:59:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 18:59:50 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 18:59:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:59:55 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 19:00:00 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 19:00:05 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 19:00:10 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 19:00:10 App: TestExipryGet got un-deployed
2021/12/16 19:00:10 No eventing-con process running
2021/12/16 19:00:10 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/16 19:00:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:00:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:00:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:00:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:00:24 Pumping document in the destination bucket for expiry get
2021/12/16 19:00:24 Starting bucket ops to source bucket
2021/12/16 19:00:25 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:00:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/16 19:00:31 Waiting for app: TestExipryGet to get deployed
2021/12/16 19:00:36 Waiting for app: TestExipryGet to get deployed
2021/12/16 19:00:41 Waiting for app: TestExipryGet to get deployed
2021/12/16 19:00:46 Waiting for app: TestExipryGet to get deployed
2021/12/16 19:00:46 App: TestExipryGet got deployed
2021/12/16 19:00:46 Starting bucket ops to source bucket
2021/12/16 19:01:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:01:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:01:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:01:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 19:01:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:01:36 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 19:01:41 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 19:01:46 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 19:01:51 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 19:01:51 App: TestExipryGet got un-deployed
2021/12/16 19:01:51 No eventing-con process running
2021/12/16 19:01:51 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/16 19:01:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:02:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:02:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:02:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:02:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:02:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (211.01s)
=== RUN   TestExipryInsert
2021/12/16 19:02:15 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:02:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/16 19:02:21 Waiting for app: TestExipryInsert to get deployed
2021/12/16 19:02:26 Waiting for app: TestExipryInsert to get deployed
2021/12/16 19:02:31 Waiting for app: TestExipryInsert to get deployed
2021/12/16 19:02:36 Waiting for app: TestExipryInsert to get deployed
2021/12/16 19:02:36 App: TestExipryInsert got deployed
2021/12/16 19:02:36 Starting bucket ops to source bucket
2021/12/16 19:03:21 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 19:03:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:03:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:03:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 19:03:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:03:26 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 19:03:31 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 19:03:36 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 19:03:41 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 19:03:41 App: TestExipryInsert got un-deployed
2021/12/16 19:03:41 No eventing-con process running
2021/12/16 19:03:41 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/16 19:03:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:03:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:03:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:03:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:03:55 Testing on the destination bucket for expiry insert
2021/12/16 19:03:55 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:03:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/16 19:04:01 Waiting for app: TestExipryInsert to get deployed
2021/12/16 19:04:06 Waiting for app: TestExipryInsert to get deployed
2021/12/16 19:04:11 Waiting for app: TestExipryInsert to get deployed
2021/12/16 19:04:16 Waiting for app: TestExipryInsert to get deployed
2021/12/16 19:04:16 App: TestExipryInsert got deployed
2021/12/16 19:04:16 Starting bucket ops to source bucket
2021/12/16 19:05:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:05:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:05:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:05:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 19:05:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:05:07 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 19:05:12 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 19:05:17 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 19:05:22 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 19:05:22 App: TestExipryInsert got un-deployed
2021/12/16 19:05:22 No eventing-con process running
2021/12/16 19:05:22 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/16 19:05:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:05:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:05:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:05:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:05:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:05:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.64s)
=== RUN   TestExipryReplace
2021/12/16 19:05:45 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:05:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 19:05:51 Waiting for app: TestExipryReplace to get deployed
2021/12/16 19:05:56 Waiting for app: TestExipryReplace to get deployed
2021/12/16 19:06:01 Waiting for app: TestExipryReplace to get deployed
2021/12/16 19:06:06 Waiting for app: TestExipryReplace to get deployed
2021/12/16 19:06:06 App: TestExipryReplace got deployed
2021/12/16 19:06:06 Starting bucket ops to source bucket
2021/12/16 19:06:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:06:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:06:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:06:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 19:06:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:06:57 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 19:07:02 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 19:07:07 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 19:07:12 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 19:07:12 App: TestExipryReplace got un-deployed
2021/12/16 19:07:12 No eventing-con process running
2021/12/16 19:07:12 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/16 19:07:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:07:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:07:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:07:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:07:26 Testing on the destination bucket for expiry insert
2021/12/16 19:07:26 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:07:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 19:07:32 Waiting for app: TestExipryReplace to get deployed
2021/12/16 19:07:37 Waiting for app: TestExipryReplace to get deployed
2021/12/16 19:07:42 Waiting for app: TestExipryReplace to get deployed
2021/12/16 19:07:47 Waiting for app: TestExipryReplace to get deployed
2021/12/16 19:07:47 App: TestExipryReplace got deployed
2021/12/16 19:07:47 Starting bucket ops to source bucket
2021/12/16 19:08:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:08:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:08:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:08:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 19:08:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:08:37 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 19:08:42 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 19:08:47 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 19:08:52 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 19:08:52 App: TestExipryReplace got un-deployed
2021/12/16 19:08:52 No eventing-con process running
2021/12/16 19:08:52 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/16 19:08:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:09:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:09:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:09:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:09:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:09:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.57s)
=== RUN   TestExipryUpsert
2021/12/16 19:09:16 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:09:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/16 19:09:22 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 19:09:27 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 19:09:32 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 19:09:37 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 19:09:37 App: TestExipryUpsert got deployed
2021/12/16 19:09:37 Starting bucket ops to source bucket
2021/12/16 19:10:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:10:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:10:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:10:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 19:10:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:10:27 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 19:10:32 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 19:10:37 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 19:10:42 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 19:10:42 App: TestExipryUpsert got un-deployed
2021/12/16 19:10:42 No eventing-con process running
2021/12/16 19:10:42 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/16 19:10:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:10:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:10:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:10:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:10:56 Testing on the destination bucket for expiry insert
2021/12/16 19:10:56 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:10:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/16 19:11:03 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 19:11:08 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 19:11:13 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 19:11:18 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 19:11:18 App: TestExipryUpsert got deployed
2021/12/16 19:11:18 Starting bucket ops to source bucket
2021/12/16 19:12:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:12:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:12:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:12:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:12:08 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 19:12:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:12:13 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 19:12:18 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 19:12:23 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 19:12:28 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 19:12:28 App: TestExipryUpsert got un-deployed
2021/12/16 19:12:28 No eventing-con process running
2021/12/16 19:12:28 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/16 19:12:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:12:40 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:12:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:12:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:12:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:12:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (215.73s)
=== RUN   TestCasReplace
2021/12/16 19:12:51 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:12:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 19:12:58 Waiting for app: TestCasReplace to get deployed
2021/12/16 19:13:03 Waiting for app: TestCasReplace to get deployed
2021/12/16 19:13:08 Waiting for app: TestCasReplace to get deployed
2021/12/16 19:13:13 Waiting for app: TestCasReplace to get deployed
2021/12/16 19:13:13 App: TestCasReplace got deployed
2021/12/16 19:13:13 Starting bucket ops to source bucket
2021/12/16 19:13:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:13:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:13:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 19:13:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:13:18 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 19:13:23 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 19:13:28 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 19:13:33 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 19:13:33 App: TestCasReplace got un-deployed
2021/12/16 19:13:33 No eventing-con process running
2021/12/16 19:13:33 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/16 19:13:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:13:45 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/16 19:13:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:13:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:13:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:13:52 Testing on the destination bucket for upsert with cas
2021/12/16 19:13:52 Starting bucket ops to source bucket
2021/12/16 19:13:52 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:13:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 19:13:58 Waiting for app: TestCasReplace to get deployed
2021/12/16 19:14:03 Waiting for app: TestCasReplace to get deployed
2021/12/16 19:14:08 Waiting for app: TestCasReplace to get deployed
2021/12/16 19:14:13 Waiting for app: TestCasReplace to get deployed
2021/12/16 19:14:13 App: TestCasReplace got deployed
2021/12/16 19:14:13 Starting bucket ops to source bucket
2021/12/16 19:14:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:14:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:14:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:14:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:14:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 19:14:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:14:28 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 19:14:33 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 19:14:38 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 19:14:43 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 19:14:43 App: TestCasReplace got un-deployed
2021/12/16 19:14:43 No eventing-con process running
2021/12/16 19:14:43 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/16 19:14:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:14:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:14:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:14:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:15:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.13s)
=== RUN   TestCasDelete
2021/12/16 19:15:02 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:15:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 19:15:08 Waiting for app: TestCasDelete to get deployed
2021/12/16 19:15:13 Waiting for app: TestCasDelete to get deployed
2021/12/16 19:15:18 Waiting for app: TestCasDelete to get deployed
2021/12/16 19:15:23 Waiting for app: TestCasDelete to get deployed
2021/12/16 19:15:23 App: TestCasDelete got deployed
2021/12/16 19:15:23 Starting bucket ops to source bucket
2021/12/16 19:15:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:15:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:15:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 19:15:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:15:28 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 19:15:33 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 19:15:38 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 19:15:43 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 19:15:43 App: TestCasDelete got un-deployed
2021/12/16 19:15:43 No eventing-con process running
2021/12/16 19:15:43 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/16 19:15:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:15:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:15:52 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:15:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:15:57 Testing on the destination bucket for delete with cas
2021/12/16 19:15:57 Starting bucket ops to source bucket
2021/12/16 19:15:57 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:15:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 19:16:03 Waiting for app: TestCasDelete to get deployed
2021/12/16 19:16:08 Waiting for app: TestCasDelete to get deployed
2021/12/16 19:16:13 Waiting for app: TestCasDelete to get deployed
2021/12/16 19:16:18 Waiting for app: TestCasDelete to get deployed
2021/12/16 19:16:18 App: TestCasDelete got deployed
2021/12/16 19:16:18 Starting bucket ops to source bucket
2021/12/16 19:16:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:16:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:16:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:16:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:16:29 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 19:16:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:16:34 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 19:16:39 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 19:16:44 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 19:16:49 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 19:16:49 App: TestCasDelete got un-deployed
2021/12/16 19:16:49 No eventing-con process running
2021/12/16 19:16:49 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/16 19:16:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:17:01 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/16 19:17:06 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/16 19:17:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:17:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:17:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.66s)
=== RUN   TestCountersIncrement
2021/12/16 19:17:12 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 19:17:18 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 19:17:19 Removing node(s): [] from the cluster
2021/12/16 19:17:24 Rebalance progress: 100
2021/12/16 19:17:24 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:17:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/16 19:17:30 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 19:17:35 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 19:17:40 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 19:17:45 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 19:17:45 App: TestCountersIncrement got deployed
2021/12/16 19:17:45 Starting bucket ops to source bucket
2021/12/16 19:17:51 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/16 19:17:56 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/16 19:17:56 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/16 19:17:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:17:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:18:01 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 19:18:06 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 19:18:11 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 19:18:16 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 19:18:16 App: TestCountersIncrement got un-deployed
2021/12/16 19:18:16 No eventing-con process running
2021/12/16 19:18:16 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/16 19:18:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:18:28 Waiting for dst bucket item count to get to: 0 curr count: 679
2021/12/16 19:18:33 Waiting for dst bucket item count to get to: 0 curr count: 166
2021/12/16 19:18:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:18:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:18:45 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 19:18:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:18:45 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:18:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/16 19:18:51 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 19:18:56 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 19:19:01 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 19:19:06 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 19:19:06 App: TestCountersIncrement got deployed
2021/12/16 19:19:06 Starting bucket ops to source bucket
2021/12/16 19:19:12 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/16 19:19:12 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/16 19:19:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:19:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:19:17 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 19:19:22 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 19:19:27 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 19:19:32 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 19:19:32 App: TestCountersIncrement got un-deployed
2021/12/16 19:19:32 No eventing-con process running
2021/12/16 19:19:32 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/16 19:19:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:19:44 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/16 19:19:49 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 19:19:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:19:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:20:01 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 19:20:06 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 19:20:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:20:06 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/16 19:20:11 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.79s)
=== RUN   TestCountersDecrement
2021/12/16 19:20:11 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 19:20:12 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 19:20:12 Removing node(s): [] from the cluster
2021/12/16 19:20:17 Rebalance progress: 100
2021/12/16 19:20:17 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:20:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/16 19:20:23 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 19:20:29 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 19:20:34 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 19:20:39 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 19:20:39 App: TestCountersDecrement got deployed
2021/12/16 19:20:39 Starting bucket ops to source bucket
2021/12/16 19:20:45 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/16 19:20:50 Waiting for dst bucket item count to get to: 1024 curr count: 322
2021/12/16 19:20:55 Waiting for dst bucket item count to get to: 1024 curr count: 322
2021/12/16 19:20:55 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/16 19:20:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:20:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:21:00 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 19:21:05 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 19:21:10 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 19:21:15 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 19:21:15 App: TestCountersDecrement got un-deployed
2021/12/16 19:21:15 No eventing-con process running
2021/12/16 19:21:15 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/16 19:21:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:21:27 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/16 19:21:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:21:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:21:38 Waiting for dst bucket item count to get to: 0 curr count: 866
2021/12/16 19:21:43 Waiting for dst bucket item count to get to: 0 curr count: 354
2021/12/16 19:21:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:21:44 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:21:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/16 19:21:50 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 19:21:55 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 19:22:00 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 19:22:05 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 19:22:05 App: TestCountersDecrement got deployed
2021/12/16 19:22:05 Starting bucket ops to source bucket
2021/12/16 19:22:11 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/16 19:22:16 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/16 19:22:16 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/16 19:22:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:22:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:22:21 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 19:22:26 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 19:22:31 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 19:22:36 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 19:22:36 App: TestCountersDecrement got un-deployed
2021/12/16 19:22:36 No eventing-con process running
2021/12/16 19:22:36 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/16 19:22:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:22:48 Waiting for dst bucket item count to get to: 0 curr count: 1349
2021/12/16 19:22:53 Waiting for dst bucket item count to get to: 0 curr count: 325
2021/12/16 19:22:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:22:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:23:05 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 19:23:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:23:05 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/16 19:23:10 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.87s)
=== RUN   TestCacheGet
2021/12/16 19:23:10 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:23:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/16 19:23:16 Waiting for app: TestCacheGet to get deployed
2021/12/16 19:23:21 Waiting for app: TestCacheGet to get deployed
2021/12/16 19:23:26 Waiting for app: TestCacheGet to get deployed
2021/12/16 19:23:31 Waiting for app: TestCacheGet to get deployed
2021/12/16 19:23:31 App: TestCacheGet got deployed
2021/12/16 19:23:31 Starting bucket ops to source bucket
2021/12/16 19:23:36 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/16 19:23:41 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/16 19:23:46 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/16 19:23:51 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/16 19:23:56 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/16 19:24:01 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/16 19:24:01 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/16 19:24:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:24:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:24:07 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 19:24:12 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 19:24:17 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 19:24:22 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 19:24:22 App: TestCacheGet got un-deployed
2021/12/16 19:24:22 No eventing-con process running
2021/12/16 19:24:22 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/16 19:24:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:24:33 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/16 19:24:38 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/16 19:24:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:24:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:24:50 Waiting for dst bucket item count to get to: 0 curr count: 14
2021/12/16 19:24:55 Waiting for dst bucket item count to get to: 0 curr count: 14
2021/12/16 19:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (105.11s)
=== RUN   TestCacheRYOW
2021/12/16 19:24:55 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:24:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/16 19:25:01 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 19:25:06 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 19:25:11 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 19:25:16 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 19:25:16 App: TestCacheRYOW got deployed
2021/12/16 19:25:16 Starting bucket ops to source bucket
2021/12/16 19:25:21 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/16 19:25:21 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/16 19:25:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:25:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:25:27 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 19:25:32 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 19:25:37 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 19:25:42 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 19:25:42 App: TestCacheRYOW got un-deployed
2021/12/16 19:25:42 No eventing-con process running
2021/12/16 19:25:42 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/16 19:25:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:25:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:25:58 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/16 19:25:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:26:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:26:10 Waiting for dst bucket item count to get to: 0 curr count: 37
2021/12/16 19:26:15 Waiting for dst bucket item count to get to: 0 curr count: 37
2021/12/16 19:26:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.23s)
=== RUN   TestCacheExpiry
2021/12/16 19:26:15 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:26:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/16 19:26:21 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 19:26:26 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 19:26:32 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 19:26:37 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 19:26:37 App: TestCacheExpiry got deployed
2021/12/16 19:26:37 Starting bucket ops to source bucket
2021/12/16 19:26:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 19:26:47 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/16 19:26:52 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/16 19:26:57 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/16 19:27:02 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/16 19:27:02 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 19:27:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:27:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:27:07 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 19:27:12 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 19:27:17 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 19:27:22 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 19:27:22 App: TestCacheExpiry got un-deployed
2021/12/16 19:27:22 No eventing-con process running
2021/12/16 19:27:22 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/16 19:27:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:27:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:27:39 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/16 19:27:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:27:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:27:50 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/16 19:27:55 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/16 19:27:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.16s)
=== RUN   TestCacheExpiryCustom
2021/12/16 19:27:55 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:27:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/16 19:28:02 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 19:28:07 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 19:28:12 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 19:28:17 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 19:28:17 App: TestCacheExpiryCustom got deployed
2021/12/16 19:28:17 Starting bucket ops to source bucket
2021/12/16 19:28:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 19:28:27 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/16 19:28:32 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/16 19:28:37 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/16 19:28:42 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/16 19:28:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 19:28:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:28:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:28:47 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 19:28:52 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 19:28:57 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 19:29:02 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 19:29:02 App: TestCacheExpiryCustom got un-deployed
2021/12/16 19:29:02 No eventing-con process running
2021/12/16 19:29:02 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/16 19:29:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:29:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:29:19 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/16 19:29:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:29:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:29:31 Waiting for dst bucket item count to get to: 0 curr count: 86
2021/12/16 19:29:36 Waiting for dst bucket item count to get to: 0 curr count: 86
2021/12/16 19:29:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.20s)
=== RUN   TestCacheOverflowCustom
2021/12/16 19:29:36 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:29:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/16 19:29:42 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 19:29:47 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 19:29:52 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 19:29:57 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 19:29:57 App: TestCacheOverflowCustom got deployed
2021/12/16 19:29:57 Starting bucket ops to source bucket
2021/12/16 19:30:02 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/16 19:30:07 Waiting for dst bucket item count to get to: 35 curr count: 6
2021/12/16 19:30:12 Waiting for dst bucket item count to get to: 35 curr count: 6
2021/12/16 19:30:17 Waiting for dst bucket item count to get to: 35 curr count: 26
2021/12/16 19:30:22 Waiting for dst bucket item count to get to: 35 curr count: 26
2021/12/16 19:30:22 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/16 19:30:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:30:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:30:27 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 19:30:32 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 19:30:37 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 19:30:42 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 19:30:42 App: TestCacheOverflowCustom got un-deployed
2021/12/16 19:30:42 No eventing-con process running
2021/12/16 19:30:42 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/16 19:30:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:30:54 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/16 19:30:59 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/16 19:30:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:31:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:31:11 Waiting for dst bucket item count to get to: 0 curr count: 34
2021/12/16 19:31:16 Waiting for dst bucket item count to get to: 0 curr count: 34
2021/12/16 19:31:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.22s)
=== RUN   TestStrictMode
2021/12/16 19:31:16 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:31:17 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/16 19:31:17 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:31:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/16 19:31:23 Waiting for app: TestDataTypes to get deployed
2021/12/16 19:31:28 Waiting for app: TestDataTypes to get deployed
2021/12/16 19:31:33 Waiting for app: TestDataTypes to get deployed
2021/12/16 19:31:38 Waiting for app: TestDataTypes to get deployed
2021/12/16 19:31:38 App: TestDataTypes got deployed
2021/12/16 19:31:38 Starting bucket ops to source bucket
2021/12/16 19:31:43 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/16 19:31:48 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/16 19:31:53 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/16 19:31:53 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/16 19:31:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:31:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:31:58 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 19:32:03 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 19:32:08 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 19:32:13 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 19:32:13 App: TestDataTypes got un-deployed
2021/12/16 19:32:13 No eventing-con process running
2021/12/16 19:32:13 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/16 19:32:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:32:25 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 19:32:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:32:37 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/16 19:32:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.07s)
=== RUN   TestEnoentGet
2021/12/16 19:32:37 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:32:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/16 19:32:43 Waiting for app: TestEnoentGet to get deployed
2021/12/16 19:32:48 Waiting for app: TestEnoentGet to get deployed
2021/12/16 19:32:53 Waiting for app: TestEnoentGet to get deployed
2021/12/16 19:32:58 Waiting for app: TestEnoentGet to get deployed
2021/12/16 19:32:58 App: TestEnoentGet got deployed
2021/12/16 19:32:58 Starting bucket ops to source bucket
2021/12/16 19:33:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 19:33:08 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 19:33:08 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 19:33:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:33:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:33:14 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 19:33:19 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 19:33:24 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 19:33:29 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 19:33:29 App: TestEnoentGet got un-deployed
2021/12/16 19:33:29 No eventing-con process running
2021/12/16 19:33:29 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/16 19:33:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:33:40 Waiting for dst bucket item count to get to: 0 curr count: 77
2021/12/16 19:33:45 Waiting for dst bucket item count to get to: 0 curr count: 77
2021/12/16 19:33:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:33:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:33:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:33:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:33:57 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:33:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/16 19:34:03 Waiting for app: TestEnoentGet to get deployed
2021/12/16 19:34:08 Waiting for app: TestEnoentGet to get deployed
2021/12/16 19:34:13 Waiting for app: TestEnoentGet to get deployed
2021/12/16 19:34:18 Waiting for app: TestEnoentGet to get deployed
2021/12/16 19:34:18 App: TestEnoentGet got deployed
2021/12/16 19:34:18 Starting bucket ops to source bucket
2021/12/16 19:34:24 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 19:34:29 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 19:34:29 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 19:34:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:34:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:34:34 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 19:34:39 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 19:34:44 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 19:34:49 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 19:34:49 App: TestEnoentGet got un-deployed
2021/12/16 19:34:49 No eventing-con process running
2021/12/16 19:34:49 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/16 19:34:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:35:01 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/16 19:35:06 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/16 19:35:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:35:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:35:17 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:35:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.37s)
=== RUN   TestEnoentDelete
2021/12/16 19:35:17 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:35:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 19:35:23 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 19:35:29 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 19:35:34 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 19:35:39 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 19:35:39 App: TestEnoentDelete got deployed
2021/12/16 19:35:39 Starting bucket ops to source bucket
2021/12/16 19:35:44 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 19:35:49 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 19:35:49 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 19:35:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:35:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:35:54 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 19:35:59 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 19:36:04 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 19:36:09 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 19:36:09 App: TestEnoentDelete got un-deployed
2021/12/16 19:36:09 No eventing-con process running
2021/12/16 19:36:09 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/16 19:36:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:36:21 Waiting for dst bucket item count to get to: 0 curr count: 87
2021/12/16 19:36:26 Waiting for dst bucket item count to get to: 0 curr count: 87
2021/12/16 19:36:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:36:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:36:37 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:36:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:36:37 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:36:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 19:36:44 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 19:36:49 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 19:36:54 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 19:36:59 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 19:36:59 App: TestEnoentDelete got deployed
2021/12/16 19:36:59 Starting bucket ops to source bucket
2021/12/16 19:37:04 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 19:37:09 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 19:37:09 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 19:37:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:37:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:37:14 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 19:37:19 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 19:37:24 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 19:37:29 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 19:37:29 App: TestEnoentDelete got un-deployed
2021/12/16 19:37:29 No eventing-con process running
2021/12/16 19:37:29 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/16 19:37:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:37:41 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/16 19:37:46 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/16 19:37:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:37:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:37:58 Waiting for dst bucket item count to get to: 0 curr count: 51
2021/12/16 19:38:03 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 19:38:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (165.31s)
=== RUN   TestError
2021/12/16 19:38:13 Waiting for app: TestError to get un-deployed
2021/12/16 19:38:13 App: TestError got un-deployed
2021/12/16 19:38:13 No eventing-con process running
2021/12/16 19:38:13 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/16 19:38:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:38:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:38:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:38:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:38:26 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:38:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/16 19:38:32 Waiting for app: TestError to get deployed
2021/12/16 19:38:37 Waiting for app: TestError to get deployed
2021/12/16 19:38:42 Waiting for app: TestError to get deployed
2021/12/16 19:38:47 Waiting for app: TestError to get deployed
2021/12/16 19:38:47 App: TestError got deployed
2021/12/16 19:38:47 Starting bucket ops to source bucket
2021/12/16 19:38:53 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/16 19:38:58 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/16 19:39:03 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/16 19:39:03 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/16 19:39:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:39:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:39:08 Waiting for app: TestError to get un-deployed
2021/12/16 19:39:13 Waiting for app: TestError to get un-deployed
2021/12/16 19:39:18 Waiting for app: TestError to get un-deployed
2021/12/16 19:39:23 Waiting for app: TestError to get un-deployed
2021/12/16 19:39:23 App: TestError got un-deployed
2021/12/16 19:39:23 No eventing-con process running
2021/12/16 19:39:23 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/16 19:39:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:39:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:39:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:39:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:39:47 Waiting for dst bucket item count to get to: 0 curr count: 300
2021/12/16 19:39:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (103.84s)
=== RUN   TestCRLF
2021/12/16 19:39:57 Waiting for app: TestCRLF to get un-deployed
2021/12/16 19:39:57 App: TestCRLF got un-deployed
2021/12/16 19:39:57 No eventing-con process running
2021/12/16 19:39:57 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/16 19:39:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:40:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:40:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:40:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:40:10 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:40:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/16 19:40:16 Waiting for app: TestCRLF to get deployed
2021/12/16 19:40:21 Waiting for app: TestCRLF to get deployed
2021/12/16 19:40:26 Waiting for app: TestCRLF to get deployed
2021/12/16 19:40:31 Waiting for app: TestCRLF to get deployed
2021/12/16 19:40:31 App: TestCRLF got deployed
2021/12/16 19:40:31 Starting bucket ops to source bucket
2021/12/16 19:40:37 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 19:40:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 19:40:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 19:40:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:40:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:40:47 Waiting for app: TestCRLF to get un-deployed
2021/12/16 19:40:52 Waiting for app: TestCRLF to get un-deployed
2021/12/16 19:40:57 Waiting for app: TestCRLF to get un-deployed
2021/12/16 19:41:02 Waiting for app: TestCRLF to get un-deployed
2021/12/16 19:41:02 App: TestCRLF got un-deployed
2021/12/16 19:41:02 No eventing-con process running
2021/12/16 19:41:02 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/16 19:41:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:41:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 19:41:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 19:41:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:41:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:41:30 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/16 19:41:35 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/16 19:41:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.81s)
=== RUN   TestImportExport
2021/12/16 19:41:45 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 19:41:45 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/16 19:41:45 No eventing-con process running
2021/12/16 19:41:45 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/16 19:41:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:41:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:41:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:41:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:42:04 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 19:42:04 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/16 19:42:04 No eventing-con process running
2021/12/16 19:42:04 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/16 19:42:06 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:42:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:42:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:42:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:42:18 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:42:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 19:42:19 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:42:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 19:42:25 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 19:42:30 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 19:42:35 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 19:42:40 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 19:42:40 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/16 19:42:45 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/16 19:42:50 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/16 19:42:55 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/16 19:42:55 App: TestImportExport_bucket_op_on_update got deployed
2021/12/16 19:42:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:43:00 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 19:43:05 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 19:43:10 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 19:43:15 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 19:43:15 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/16 19:43:15 No eventing-con process running
2021/12/16 19:43:15 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/16 19:43:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:43:20 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 19:43:25 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 19:43:30 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 19:43:35 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 19:43:35 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/16 19:43:35 No eventing-con process running
2021/12/16 19:43:36 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/16 19:43:53 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 19:43:53 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/16 19:43:53 No eventing-con process running
2021/12/16 19:43:53 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/16 19:43:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:44:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:44:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:44:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:44:11 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 19:44:11 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/16 19:44:11 No eventing-con process running
2021/12/16 19:44:12 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/16 19:44:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:44:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:44:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:44:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.74s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/16 19:44:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:44:35 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 19:44:35 No eventing-con process running
2021/12/16 19:44:35 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/16 19:44:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:44:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:44:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:44:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:44:49 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:44:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 19:44:50 Starting bucket ops to source bucket
2021/12/16 19:44:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:45:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:45:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:45:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:45:18 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 19:45:18 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 19:45:23 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 19:45:23 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 19:45:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:45:23 Undeploying app: bucket_op_on_update
2021/12/16 19:45:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:45:28 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:45:33 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:45:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:45:43 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:45:43 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 19:45:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:45:43 No eventing-con process running
2021/12/16 19:45:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:45:52 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:46:27 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:46:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 19:46:28 Starting bucket ops to source bucket
2021/12/16 19:46:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:46:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:46:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:46:51 Waiting for dst bucket item count to get to: 5000 curr count: 22
2021/12/16 19:46:56 Waiting for dst bucket item count to get to: 5000 curr count: 2522
2021/12/16 19:46:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 19:47:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 19:47:01 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 19:47:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:47:01 Undeploying app: bucket_op_on_update
2021/12/16 19:47:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:47:06 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:47:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:47:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:47:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:47:22 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 19:47:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:47:22 No eventing-con process running
2021/12/16 19:47:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:47:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:48:06 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:48:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 19:48:07 Starting bucket ops to source bucket
2021/12/16 19:48:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:48:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:48:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:48:30 Waiting for dst bucket item count to get to: 5000 curr count: 1400
2021/12/16 19:48:35 Waiting for dst bucket item count to get to: 5000 curr count: 1400
2021/12/16 19:48:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 19:48:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 19:48:40 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 19:48:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:48:40 Undeploying app: bucket_op_on_update
2021/12/16 19:48:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:48:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:48:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:48:55 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:49:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:49:00 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 19:49:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:49:00 No eventing-con process running
2021/12/16 19:49:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:49:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:49:44 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:49:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 19:49:45 Starting bucket ops to source bucket
2021/12/16 19:49:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:49:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:50:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:50:08 Waiting for dst bucket item count to get to: 5000 curr count: 2709
2021/12/16 19:50:13 Waiting for dst bucket item count to get to: 5000 curr count: 2709
2021/12/16 19:50:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 19:50:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 19:50:18 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 19:50:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:50:18 Undeploying app: bucket_op_on_update
2021/12/16 19:50:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:50:23 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:50:28 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:50:33 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:50:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:50:38 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 19:50:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:50:38 No eventing-con process running
2021/12/16 19:50:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:50:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:51:22 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:51:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 19:51:24 Starting bucket ops to source bucket
2021/12/16 19:51:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:51:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:51:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:51:47 Waiting for dst bucket item count to get to: 5000 curr count: 2197
2021/12/16 19:51:52 Waiting for dst bucket item count to get to: 5000 curr count: 4697
2021/12/16 19:51:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 19:51:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 19:51:57 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 19:51:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:51:57 Undeploying app: bucket_op_on_update
2021/12/16 19:51:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:52:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:52:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:52:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:52:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 19:52:17 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 19:52:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:52:17 No eventing-con process running
2021/12/16 19:52:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:52:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:53:01 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (515.82s)
=== RUN   TestOnUpdateN1QLOp
2021/12/16 19:53:11 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 19:53:11 App: TestOnUpdateN1QLOp got un-deployed
2021/12/16 19:53:11 No eventing-con process running
2021/12/16 19:53:11 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/16 19:53:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:53:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:53:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:53:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:53:24 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:53:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/16 19:53:26 Starting bucket ops to source bucket
2021/12/16 19:53:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:53:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:53:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:53:49 Waiting for dst bucket item count to get to: 5000 curr count: 979
2021/12/16 19:53:54 Waiting for dst bucket item count to get to: 5000 curr count: 979
2021/12/16 19:53:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 19:53:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:53:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:53:59 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 19:54:04 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 19:54:09 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 19:54:14 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 19:54:14 App: TestOnUpdateN1QLOp got un-deployed
2021/12/16 19:54:14 No eventing-con process running
2021/12/16 19:54:14 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/16 19:54:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:54:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 19:54:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:54:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:54:37 Waiting for dst bucket item count to get to: 0 curr count: 2520
2021/12/16 19:54:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (96.53s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/16 19:54:47 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 19:54:47 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/16 19:54:47 No eventing-con process running
2021/12/16 19:54:47 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/16 19:54:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:54:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:54:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:55:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:55:01 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:55:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 19:55:02 Starting bucket ops to source bucket
2021/12/16 19:55:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:55:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:55:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:55:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:55:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 19:55:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:55:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:55:30 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 19:55:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 19:55:40 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 19:55:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 19:55:45 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/16 19:55:45 No eventing-con process running
2021/12/16 19:55:45 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/16 19:55:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:55:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 19:55:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:55:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:56:09 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 19:56:14 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 19:56:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.66s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/16 19:56:24 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 19:56:24 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 19:56:24 No eventing-con process running
2021/12/16 19:56:24 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/16 19:56:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:56:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:56:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:56:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:56:38 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:56:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 19:56:39 Starting bucket ops to source bucket
2021/12/16 19:56:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:56:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:56:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:57:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:57:07 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 19:57:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 19:57:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:57:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:57:12 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 19:57:17 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 19:57:22 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 19:57:27 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 19:57:27 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 19:57:27 No eventing-con process running
2021/12/16 19:57:27 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/16 19:57:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:57:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 19:57:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 19:57:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:57:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:57:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 19:57:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.66s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/16 19:58:06 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 19:58:06 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/16 19:58:06 No eventing-con process running
2021/12/16 19:58:06 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/16 19:58:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:58:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:58:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:58:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:58:19 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:58:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/16 19:58:21 Starting bucket ops to source bucket
2021/12/16 19:58:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:58:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:58:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:58:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 19:58:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 19:58:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 19:58:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 19:58:49 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 19:58:54 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 19:58:59 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 19:59:04 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 19:59:04 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/16 19:59:04 No eventing-con process running
2021/12/16 19:59:04 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/16 19:59:06 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:59:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 19:59:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:59:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:59:27 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 19:59:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (91.71s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/16 19:59:37 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 19:59:37 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/16 19:59:37 No eventing-con process running
2021/12/16 19:59:37 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/16 19:59:39 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:59:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:59:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 19:59:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 19:59:51 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/16 19:59:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/16 19:59:53 Starting bucket ops to source bucket
2021/12/16 20:00:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:00:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:00:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:00:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:00:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:00:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:00:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:00:21 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 20:00:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 20:00:31 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 20:00:36 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 20:00:36 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/16 20:00:36 No eventing-con process running
2021/12/16 20:00:36 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/16 20:00:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:00:48 Waiting for dst bucket item count to get to: 0 curr count: 3057
2021/12/16 20:00:53 Waiting for dst bucket item count to get to: 0 curr count: 557
2021/12/16 20:00:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:00:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:01:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:01:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.26s)
=== RUN   TestOnDeleteBucketOp
2021/12/16 20:01:15 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 20:01:15 App: TestOnDeleteBucketOp got un-deployed
2021/12/16 20:01:15 No eventing-con process running
2021/12/16 20:01:15 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/16 20:01:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:01:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:01:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:01:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:01:28 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:01:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 20:01:29 Starting bucket ops to source bucket
2021/12/16 20:01:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:01:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:01:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:01:55 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/16 20:01:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:01:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:01:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:02:01 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 20:02:06 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 20:02:11 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 20:02:16 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 20:02:16 App: TestOnDeleteBucketOp got un-deployed
2021/12/16 20:02:16 No eventing-con process running
2021/12/16 20:02:16 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/16 20:02:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:02:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:02:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:02:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:02:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (90.00s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/16 20:02:45 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 20:02:45 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/16 20:02:45 No eventing-con process running
2021/12/16 20:02:45 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/16 20:02:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:02:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:02:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:02:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:02:58 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:02:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/16 20:02:59 Starting bucket ops to source bucket
2021/12/16 20:03:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:03:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:03:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:03:25 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/16 20:03:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:03:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:03:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:03:30 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 20:03:36 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 20:03:41 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 20:03:46 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 20:03:46 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/16 20:03:46 No eventing-con process running
2021/12/16 20:03:46 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/16 20:03:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:03:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:03:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:04:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:04:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.94s)
=== RUN   TestTimerBucketOp
2021/12/16 20:04:15 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 20:04:15 App: TestTimerBucketOp got un-deployed
2021/12/16 20:04:15 No eventing-con process running
2021/12/16 20:04:15 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/16 20:04:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:04:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:04:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:04:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:04:28 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:04:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 20:04:29 Starting bucket ops to source bucket
2021/12/16 20:04:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:04:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:04:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:04:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:04:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:05:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:05:07 Waiting for dst bucket item count to get to: 5000 curr count: 1486
2021/12/16 20:05:12 Waiting for dst bucket item count to get to: 5000 curr count: 3986
2021/12/16 20:05:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:05:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:05:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:05:17 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 20:05:22 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 20:05:27 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 20:05:32 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 20:05:33 App: TestTimerBucketOp got un-deployed
2021/12/16 20:05:33 No eventing-con process running
2021/12/16 20:05:33 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/16 20:05:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:05:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:05:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 20:05:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:05:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:06:01 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:06:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:06:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (121.44s)
=== RUN   TestTimerInPastBucketOp
2021/12/16 20:06:16 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 20:06:16 App: TestTimerInPastBucketOp got un-deployed
2021/12/16 20:06:16 No eventing-con process running
2021/12/16 20:06:16 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/16 20:06:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:06:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:06:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:06:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:06:30 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:06:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 20:06:31 Starting bucket ops to source bucket
2021/12/16 20:06:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:06:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:06:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:06:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:06:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:07:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:07:09 Waiting for dst bucket item count to get to: 5000 curr count: 2233
2021/12/16 20:07:14 Waiting for dst bucket item count to get to: 5000 curr count: 2233
2021/12/16 20:07:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:07:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:07:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:07:19 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 20:07:24 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 20:07:29 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 20:07:34 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 20:07:34 App: TestTimerInPastBucketOp got un-deployed
2021/12/16 20:07:34 No eventing-con process running
2021/12/16 20:07:34 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/16 20:07:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:07:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:07:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:07:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:07:58 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 20:07:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.48s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/16 20:08:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:08:08 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 20:08:08 No eventing-con process running
2021/12/16 20:08:08 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/16 20:08:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:08:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:08:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:08:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:08:21 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:08:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 20:08:22 Starting bucket ops to source bucket
2021/12/16 20:08:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:08:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:08:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:08:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:08:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:08:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:09:00 Waiting for dst bucket item count to get to: 5000 curr count: 745
2021/12/16 20:09:05 Waiting for dst bucket item count to get to: 5000 curr count: 745
2021/12/16 20:09:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:09:10 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 20:09:10 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 20:09:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:09:10 Undeploying app: bucket_op_with_timer
2021/12/16 20:09:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:09:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:09:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:09:25 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:09:31 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:09:31 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 20:09:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:09:31 No eventing-con process running
2021/12/16 20:09:32 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:09:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:10:15 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:10:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 20:10:16 Starting bucket ops to source bucket
2021/12/16 20:10:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:10:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:10:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:10:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:10:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:10:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:10:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:10:59 Waiting for dst bucket item count to get to: 5000 curr count: 4964
2021/12/16 20:11:04 Waiting for dst bucket item count to get to: 5000 curr count: 4964
2021/12/16 20:11:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:11:09 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 20:11:09 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 20:11:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:11:09 Undeploying app: bucket_op_with_timer
2021/12/16 20:11:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:11:14 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:11:19 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:11:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:11:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:11:29 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 20:11:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:11:29 No eventing-con process running
2021/12/16 20:11:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:11:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:12:13 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:12:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 20:12:14 Starting bucket ops to source bucket
2021/12/16 20:12:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:12:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:12:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:12:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:12:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:12:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:12:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:12:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 20:12:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:13:02 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 20:13:02 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 20:13:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:13:02 Undeploying app: bucket_op_with_timer
2021/12/16 20:13:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:13:07 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:13:12 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:13:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:13:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:13:22 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 20:13:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:13:23 No eventing-con process running
2021/12/16 20:13:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:13:32 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:14:07 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:14:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 20:14:08 Starting bucket ops to source bucket
2021/12/16 20:14:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:14:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:14:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:14:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:14:36 Waiting for dst bucket item count to get to: 5000 curr count: 95
2021/12/16 20:14:41 Waiting for dst bucket item count to get to: 5000 curr count: 777
2021/12/16 20:14:46 Waiting for dst bucket item count to get to: 5000 curr count: 3182
2021/12/16 20:14:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:14:51 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 20:14:51 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 20:14:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:14:51 Undeploying app: bucket_op_with_timer
2021/12/16 20:14:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:14:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:15:01 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:15:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:15:11 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:15:11 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 20:15:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:15:11 No eventing-con process running
2021/12/16 20:15:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:15:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:15:55 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:15:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 20:15:56 Starting bucket ops to source bucket
2021/12/16 20:16:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:16:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:16:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:16:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:16:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:16:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:16:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:16:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:16:39 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 20:16:39 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 20:16:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:16:39 Undeploying app: bucket_op_with_timer
2021/12/16 20:16:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:16:44 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:16:49 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:16:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:16:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 20:16:59 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 20:16:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:16:59 No eventing-con process running
2021/12/16 20:17:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:17:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:17:43 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (585.89s)
=== RUN   TestMultipleHandlers
2021/12/16 20:17:53 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 20:17:53 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/16 20:17:53 No eventing-con process running
2021/12/16 20:17:53 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/16 20:17:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:18:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:18:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:18:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:18:12 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 20:18:12 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/16 20:18:12 No eventing-con process running
2021/12/16 20:18:12 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/16 20:18:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:18:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:18:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:18:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:18:26 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:18:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 20:18:27 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:18:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/16 20:18:33 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 20:18:38 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 20:18:43 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 20:18:48 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 20:18:48 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/16 20:18:53 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/16 20:18:58 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/16 20:19:03 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/16 20:19:03 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/16 20:19:03 Starting bucket ops to source bucket
2021/12/16 20:19:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:19:18 Waiting for dst bucket item count to get to: 5000 curr count: 4535
2021/12/16 20:19:23 Waiting for dst bucket item count to get to: 5000 curr count: 4535
2021/12/16 20:19:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:19:23 Starting bucket ops to source bucket
2021/12/16 20:19:38 Waiting for dst bucket item count to get to: 10000 curr count: 5622
2021/12/16 20:19:43 Waiting for dst bucket item count to get to: 10000 curr count: 5622
2021/12/16 20:19:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 20:19:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:19:43 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:19:43 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:19:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:19:49 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 20:19:54 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 20:19:59 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 20:20:04 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 20:20:04 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/16 20:20:04 No eventing-con process running
2021/12/16 20:20:04 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/16 20:20:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:20:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:20:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:20:23 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 20:20:28 Waiting for dst bucket item count to get to: 0 curr count: 4996
2021/12/16 20:20:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:20:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:20:33 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 20:20:38 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 20:20:43 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 20:20:48 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 20:20:48 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/16 20:20:48 No eventing-con process running
2021/12/16 20:20:48 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/16 20:20:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:20:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:21:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (198.02s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/16 20:21:11 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 20:21:11 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/16 20:21:12 No eventing-con process running
2021/12/16 20:21:12 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/16 20:21:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:21:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:21:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:21:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:21:25 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:21:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 20:21:26 Starting bucket ops to source bucket
2021/12/16 20:21:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:21:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:21:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:21:49 Waiting for dst bucket item count to get to: 5000 curr count: 1991
2021/12/16 20:21:54 Waiting for dst bucket item count to get to: 5000 curr count: 1991
2021/12/16 20:21:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:21:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:21:59 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 20:21:59 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 20:21:59 Pausing app: bucket_op_on_update
2021/12/16 20:21:59 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:22:04 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 20:22:04 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 20:22:04 Resuming app: bucket_op_on_update
2021/12/16 20:22:06 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:22:06 Starting bucket ops to source bucket
2021/12/16 20:22:13 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:22:18 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:22:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:22:24 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 20:22:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:22:29 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 20:22:29 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 20:22:29 Pausing app: bucket_op_on_update
2021/12/16 20:22:29 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:22:34 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 20:22:34 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 20:22:34 Resuming app: bucket_op_on_update
2021/12/16 20:22:35 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:22:35 Starting bucket ops to source bucket
2021/12/16 20:22:43 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 20:22:48 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 20:22:53 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 20:22:53 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/16 20:22:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:22:58 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 20:22:58 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 20:22:58 Pausing app: bucket_op_on_update
2021/12/16 20:22:58 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:23:03 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 20:23:03 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 20:23:03 Resuming app: bucket_op_on_update
2021/12/16 20:23:04 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:23:04 Starting bucket ops to source bucket
2021/12/16 20:23:12 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 20:23:17 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 20:23:22 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 20:23:22 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/16 20:23:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:23:27 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 20:23:27 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 20:23:27 Pausing app: bucket_op_on_update
2021/12/16 20:23:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:23:32 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 20:23:32 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 20:23:32 Resuming app: bucket_op_on_update
2021/12/16 20:23:34 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:23:34 Starting bucket ops to source bucket
2021/12/16 20:23:41 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 20:23:46 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 20:23:51 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 20:23:56 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 20:23:56 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/16 20:23:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:24:02 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 20:24:02 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 20:24:02 Pausing app: bucket_op_on_update
2021/12/16 20:24:02 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:24:07 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 20:24:07 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 20:24:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:24:12 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 20:24:17 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 20:24:22 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 20:24:27 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 20:24:27 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/16 20:24:27 No eventing-con process running
2021/12/16 20:24:27 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/16 20:24:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:24:39 Waiting for dst bucket item count to get to: 0 curr count: 22406
2021/12/16 20:24:44 Waiting for dst bucket item count to get to: 0 curr count: 9906
2021/12/16 20:24:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:24:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:24:55 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 20:24:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (234.02s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/16 20:25:06 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 20:25:06 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/16 20:25:06 No eventing-con process running
2021/12/16 20:25:06 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/16 20:25:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:25:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:25:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:25:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:25:19 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:25:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 20:25:20 Starting bucket ops to source bucket
2021/12/16 20:25:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:25:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:25:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:25:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:25:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:25:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:25:58 Waiting for dst bucket item count to get to: 5000 curr count: 3989
2021/12/16 20:26:03 Waiting for dst bucket item count to get to: 5000 curr count: 3989
2021/12/16 20:26:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:26:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:26:08 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 20:26:08 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 20:26:08 Pausing app: bucket_op_with_timer
2021/12/16 20:26:08 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:26:13 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 20:26:13 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 20:26:13 Resuming app: bucket_op_with_timer
2021/12/16 20:26:14 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:26:14 Starting bucket ops to source bucket
2021/12/16 20:26:23 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:26:28 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:26:33 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:26:38 Waiting for dst bucket item count to get to: 10000 curr count: 5435
2021/12/16 20:26:43 Waiting for dst bucket item count to get to: 10000 curr count: 5435
2021/12/16 20:26:48 Waiting for dst bucket item count to get to: 10000 curr count: 9900
2021/12/16 20:26:53 Waiting for dst bucket item count to get to: 10000 curr count: 9900
2021/12/16 20:26:53 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 20:26:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:26:58 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 20:26:58 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 20:26:58 Pausing app: bucket_op_with_timer
2021/12/16 20:26:58 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:27:03 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 20:27:03 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 20:27:03 Resuming app: bucket_op_with_timer
2021/12/16 20:27:04 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:27:04 Starting bucket ops to source bucket
2021/12/16 20:27:12 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 20:27:17 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 20:27:22 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 20:27:27 Waiting for dst bucket item count to get to: 15000 curr count: 10311
2021/12/16 20:27:32 Waiting for dst bucket item count to get to: 15000 curr count: 10311
2021/12/16 20:27:32 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/16 20:27:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:27:37 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 20:27:37 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 20:27:37 Pausing app: bucket_op_with_timer
2021/12/16 20:27:37 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:27:42 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 20:27:42 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 20:27:42 Resuming app: bucket_op_with_timer
2021/12/16 20:27:43 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:27:43 Starting bucket ops to source bucket
2021/12/16 20:27:51 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 20:27:56 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 20:28:01 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 20:28:06 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 20:28:11 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 20:28:16 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 20:28:21 Waiting for dst bucket item count to get to: 20000 curr count: 19357
2021/12/16 20:28:26 Waiting for dst bucket item count to get to: 20000 curr count: 19357
2021/12/16 20:28:26 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/16 20:28:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:28:31 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 20:28:31 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 20:28:31 Pausing app: bucket_op_with_timer
2021/12/16 20:28:32 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:28:37 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 20:28:37 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 20:28:37 Resuming app: bucket_op_with_timer
2021/12/16 20:28:38 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:28:38 Starting bucket ops to source bucket
2021/12/16 20:28:46 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 20:28:51 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 20:28:56 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 20:29:01 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 20:29:06 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 20:29:11 Waiting for dst bucket item count to get to: 25000 curr count: 20391
2021/12/16 20:29:16 Waiting for dst bucket item count to get to: 25000 curr count: 22891
2021/12/16 20:29:16 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/16 20:29:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:29:21 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 20:29:21 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 20:29:21 Pausing app: bucket_op_with_timer
2021/12/16 20:29:21 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:29:26 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 20:29:26 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 20:29:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:29:31 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 20:29:36 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 20:29:41 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 20:29:46 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 20:29:46 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/16 20:29:46 No eventing-con process running
2021/12/16 20:29:46 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/16 20:29:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:29:58 Waiting for dst bucket item count to get to: 0 curr count: 14713
2021/12/16 20:30:03 Waiting for dst bucket item count to get to: 0 curr count: 2213
2021/12/16 20:30:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:30:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:30:15 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 20:30:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.15s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/16 20:30:25 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 20:30:25 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/16 20:30:25 No eventing-con process running
2021/12/16 20:30:25 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/16 20:30:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:30:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:30:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:30:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:30:38 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:30:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/16 20:30:44 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 20:30:49 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 20:30:54 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 20:30:59 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 20:30:59 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/16 20:30:59 Starting bucket ops to source bucket
2021/12/16 20:31:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:31:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:31:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:31:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:31:29 Waiting for dst bucket item count to get to: 5000 curr count: 1225
2021/12/16 20:31:34 Waiting for dst bucket item count to get to: 5000 curr count: 2452
2021/12/16 20:31:39 Waiting for dst bucket item count to get to: 5000 curr count: 3727
2021/12/16 20:31:39 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:31:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:31:39 Pausing app: bucket_op_with_timer
2021/12/16 20:31:39 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:31:44 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/16 20:31:44 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/16 20:31:44 Resuming app: bucket_op_with_timer
2021/12/16 20:31:45 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:31:45 Starting bucket ops to source bucket
2021/12/16 20:31:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:32:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:32:07 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:32:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:32:17 Waiting for dst bucket item count to get to: 10000 curr count: 7358
2021/12/16 20:32:22 Waiting for dst bucket item count to get to: 10000 curr count: 7358
2021/12/16 20:32:22 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 20:32:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:32:27 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 20:32:32 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 20:32:37 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 20:32:42 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 20:32:42 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/16 20:32:42 No eventing-con process running
2021/12/16 20:32:42 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/16 20:32:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:32:54 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 20:32:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:32:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:33:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:33:10 Waiting for dst bucket item count to get to: 0 curr count: 7931
2021/12/16 20:33:15 Waiting for dst bucket item count to get to: 0 curr count: 7931
2021/12/16 20:33:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (180.74s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/16 20:33:25 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 20:33:25 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/16 20:33:25 No eventing-con process running
2021/12/16 20:33:25 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/16 20:33:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:33:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:33:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:33:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:33:39 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:33:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/16 20:33:45 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 20:33:50 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 20:33:55 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 20:34:00 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 20:34:00 App: TestPauseResumeWithEventingReb got deployed
2021/12/16 20:34:00 Starting bucket ops to source bucket
2021/12/16 20:34:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:34:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:34:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:34:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:34:30 Waiting for dst bucket item count to get to: 5000 curr count: 3804
2021/12/16 20:34:35 Waiting for dst bucket item count to get to: 5000 curr count: 3804
2021/12/16 20:34:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:34:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:34:35 Pausing app: bucket_op_with_timer
2021/12/16 20:34:35 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:34:40 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/16 20:34:40 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/16 20:34:40 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 20:34:40 Removing node(s): [] from the cluster
2021/12/16 20:34:45 Rebalance progress: 29.16666666666666
2021/12/16 20:34:50 Rebalance progress: 29.16666666666666
2021/12/16 20:34:55 Rebalance progress: 100
2021/12/16 20:34:55 Resuming app: bucket_op_with_timer
2021/12/16 20:34:56 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:34:56 Starting bucket ops to source bucket
2021/12/16 20:35:08 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:35:13 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:35:18 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:35:23 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 20:35:28 Waiting for dst bucket item count to get to: 10000 curr count: 5982
2021/12/16 20:35:33 Waiting for dst bucket item count to get to: 10000 curr count: 5982
2021/12/16 20:35:33 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 20:35:33 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 20:35:38 Rebalance progress: 29.16666666666666
2021/12/16 20:35:43 Rebalance progress: 29.16666666666666
2021/12/16 20:35:48 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/16 20:35:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:78} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 20:36:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:78} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 20:36:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:78} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 20:36:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:78} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 20:36:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:78} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 20:37:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:78} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 20:37:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:78} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 20:37:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:78} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 20:37:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:78} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 06dee0ebd7f756acf894cdd964b016a4
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
Producer node: 172.31.2.134:9300	Node UUID: 072fe370342ec7975c4d2c9c9e297d1d
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
Producer node: 127.0.0.1:9303	Node UUID: 40a17e08bf3c4769379a67768846dd4e
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/16 20:38:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:38:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:38:14 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 20:38:19 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 20:38:24 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 20:38:29 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 20:38:29 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/16 20:38:29 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:38:29 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/16 20:38:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:38:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8425
2021/12/16 20:38:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8425
2021/12/16 20:38:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:38:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:38:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:38:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (342.01s)
=== RUN   TestBucketDeleteAfterPause
2021/12/16 20:39:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/16 20:39:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/16 20:39:07 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:39:07 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/16 20:39:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:39:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:39:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:39:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:39:21 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:39:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/16 20:39:27 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 20:39:32 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 20:39:37 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 20:39:42 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 20:39:42 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/16 20:39:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 20:39:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:39:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:39:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:39:52 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/16 20:39:52 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:39:57 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/16 20:39:57 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/16 20:39:57 handler_test.go:716: Deleting source bucket: default
2021/12/16 20:40:03 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/16 20:40:03 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/16 20:40:08 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/16 20:40:08 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/16 20:40:08 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:40:08 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/16 20:40:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:40:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:40:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:40:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:40:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (89.20s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/16 20:40:37 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 20:40:37 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/16 20:40:37 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:40:37 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/16 20:40:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:40:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:40:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:40:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:40:50 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:40:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/16 20:40:56 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 20:41:01 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 20:41:06 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 20:41:11 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 20:41:11 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/16 20:41:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 20:41:46 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/16 20:41:46 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:41:51 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/16 20:41:51 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/16 20:44:51 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/16 20:44:51 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/16 20:44:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/16 20:44:57 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 20:44:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:44:57 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/16 20:45:02 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 20:45:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:45:02 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3361 expected 5000
2021/12/16 20:45:07 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 20:45:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:45:07 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 4449 expected 5000
2021/12/16 20:45:12 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 20:45:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:45:12 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/16 20:45:12 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/16 20:45:12 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:45:22 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 20:45:27 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 20:45:32 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 20:45:32 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/16 20:45:32 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:45:32 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/16 20:45:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:45:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:45:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 20:45:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:45:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:45:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.88s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/16 20:46:05 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 20:46:05 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/16 20:46:05 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:46:06 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/16 20:46:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:46:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:46:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:46:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:46:19 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:46:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 20:46:25 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 20:46:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 20:46:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 20:46:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 20:46:40 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/16 20:46:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 20:46:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:46:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:46:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 20:46:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:46:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:46:59 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/16 20:46:59 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:47:29 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/16 20:47:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 20:47:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 20:47:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 20:47:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 20:47:50 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 20:47:50 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/16 20:47:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 20:48:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:48:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 341
2021/12/16 20:48:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 341
2021/12/16 20:48:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:48:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:48:13 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/16 20:48:13 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:48:23 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 20:48:28 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 20:48:33 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 20:48:33 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/16 20:48:33 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:48:33 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/16 20:48:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:48:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:48:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:48:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (170.48s)
=== RUN   TestCPPWorkerCleanup
2021/12/16 20:48:56 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 20:48:56 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/16 20:48:56 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:48:56 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/16 20:48:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:49:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:49:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:49:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:49:09 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:49:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/16 20:49:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 20:49:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:49:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:49:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:49:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:49:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4817
2021/12/16 20:49:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4817
2021/12/16 20:49:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:49:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:49:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:49:48 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 20:49:53 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 20:49:58 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 20:50:03 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 20:50:03 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/16 20:50:03 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:50:03 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/16 20:50:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:50:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:50:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 20:50:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:50:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:50:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4424
2021/12/16 20:50:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4424
2021/12/16 20:50:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (139.78s)
=== RUN   TestWithUserXattrs
2021/12/16 20:51:16 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 20:51:16 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/16 20:51:16 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:51:16 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/16 20:51:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:51:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:51:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:51:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:51:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 20:51:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 20:51:33 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:51:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/16 20:51:39 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 20:51:44 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 20:51:49 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 20:51:54 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 20:51:54 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/16 20:51:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 20:52:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:52:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4229
2021/12/16 20:52:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4229
2021/12/16 20:52:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:52:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:52:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 20:52:19 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 20:52:24 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 20:52:29 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 20:52:34 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 20:52:34 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/16 20:52:34 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:52:34 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/16 20:52:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:52:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:52:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:52:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:52:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:52:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.09s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/16 20:53:07 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 20:53:07 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/16 20:53:07 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:53:07 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/16 20:53:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:53:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:53:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:53:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:53:20 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:53:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/16 20:53:26 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 20:53:31 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 20:53:36 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 20:53:41 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 20:53:41 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/16 20:53:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 20:53:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:53:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 20:53:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 20:53:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:53:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:54:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3326
2021/12/16 20:54:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 826
2021/12/16 20:54:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:54:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:54:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 20:54:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:54:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 20:54:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 14497
2021/12/16 20:54:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 14497
2021/12/16 20:54:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/16 20:54:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 20:54:57 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 20:55:02 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 20:55:07 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 20:55:12 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 20:55:17 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 20:55:17 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/16 20:55:17 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:55:22 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 20:55:22 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/16 20:55:22 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:55:22 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/16 20:55:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:55:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 20:55:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/16 20:55:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:55:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:55:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 17334
2021/12/16 20:55:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 17334
2021/12/16 20:55:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (178.31s)
=== RUN   TestMetaBucketDelete
2021/12/16 20:56:05 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/16 20:56:05 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/16 20:56:05 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:56:05 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/16 20:56:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:56:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:56:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:56:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:56:18 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:56:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 20:56:24 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 20:56:29 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 20:56:34 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 20:56:39 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 20:56:39 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/16 20:56:39 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/16 20:56:41 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/16 20:56:46 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/16 20:56:46 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/16 20:57:01 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/16 20:57:01 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/16 20:57:01 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:57:01 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/16 20:57:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:57:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:57:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:57:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.99s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/16 20:57:24 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 20:57:24 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 20:57:24 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:57:24 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/16 20:57:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:57:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:57:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:57:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:57:37 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:57:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 20:57:57 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/16 20:57:58 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/16 20:58:13 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 20:58:13 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 20:58:28 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 20:58:28 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 20:58:28 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:58:28 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/16 20:58:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:58:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:58:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:58:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.77s)
=== RUN   TestSourceBucketDelete
2021/12/16 20:58:51 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/16 20:58:51 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/16 20:58:51 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:58:51 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/16 20:58:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:58:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:58:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:59:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:59:04 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 20:59:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 20:59:10 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 20:59:15 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 20:59:20 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 20:59:25 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 20:59:25 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/16 20:59:25 handler_test.go:945: Deleting source bucket: default
2021/12/16 20:59:27 handler_test.go:947: Deleted source bucket: default
2021/12/16 20:59:32 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/16 20:59:32 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/16 20:59:47 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/16 20:59:47 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/16 20:59:47 eventingSetup.go:46: No eventing-con process running
2021/12/16 20:59:47 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/16 20:59:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 20:59:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 20:59:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:00:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.13s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/16 21:00:10 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/16 21:00:10 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/16 21:00:10 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:00:10 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/16 21:00:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:00:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:00:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:00:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:00:23 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:00:24 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 21:00:43 handler_test.go:966: Deleting source bucket: default
2021/12/16 21:00:44 handler_test.go:968: Deleted source bucket: default
2021/12/16 21:00:59 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/16 21:00:59 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/16 21:01:14 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/16 21:01:15 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/16 21:01:15 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:01:15 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/16 21:01:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:01:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:01:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:01:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.63s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/16 21:01:38 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 21:01:38 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 21:01:38 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:01:38 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/16 21:01:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:01:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:01:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:01:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:01:51 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:01:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 21:02:11 handler_test.go:988: Deleting source bucket: default
2021/12/16 21:02:12 handler_test.go:990: Deleted source bucket: default
2021/12/16 21:02:12 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/16 21:02:13 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/16 21:02:28 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 21:02:28 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 21:02:43 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 21:02:43 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 21:02:43 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:02:43 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/16 21:02:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:02:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:02:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:02:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.25s)
=== RUN   TestUndeployDuringBootstrap
2021/12/16 21:03:06 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 21:03:06 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/16 21:03:06 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:03:06 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/16 21:03:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:03:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:03:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:03:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:03:19 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:03:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 21:03:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 21:03:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:03:33 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:03:33 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/16 21:03:33 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/16 21:03:33 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/16 21:03:38 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/16 21:03:43 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/16 21:03:48 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 21:03:53 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 21:03:58 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 21:03:58 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/16 21:03:58 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:04:23 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 21:04:23 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/16 21:04:23 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:04:23 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/16 21:04:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:04:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 21:04:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:04:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:04:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3800
2021/12/16 21:04:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (110.29s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/16 21:04:51 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:04:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/16 21:04:57 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 21:05:02 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 21:05:07 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 21:05:12 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 21:05:12 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/16 21:05:12 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/16 21:05:12 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 21:05:13 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/16 21:05:18 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/16 21:05:18 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/16 21:05:18 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:05:18 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/16 21:05:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:05:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:05:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:05:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.66s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/16 21:05:41 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 21:05:41 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/16 21:05:41 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:05:41 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/16 21:05:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:05:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:05:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:05:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:05:54 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:05:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/16 21:06:00 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 21:06:05 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 21:06:10 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 21:06:15 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 21:06:15 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/16 21:06:15 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 21:06:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:06:51 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 21:06:56 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 21:07:01 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 21:07:06 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 21:07:06 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/16 21:07:06 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:08:51 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 21:08:51 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/16 21:08:51 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:08:51 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/16 21:08:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:09:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 21:09:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 21:09:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:09:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:09:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4964
2021/12/16 21:09:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2464
2021/12/16 21:09:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (233.03s)
=== RUN   TestUndeployWithKVFailover
2021/12/16 21:09:34 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/16 21:09:34 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/16 21:09:34 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:09:34 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/16 21:09:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:09:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:09:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:09:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:09:47 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:09:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/16 21:09:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 21:09:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:10:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:10:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:10:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:10:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:10:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 117
2021/12/16 21:10:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2617
2021/12/16 21:10:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 21:10:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 21:10:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 21:10:31 Rebalance progress: 29.16666666666666
2021/12/16 21:10:36 Rebalance progress: 29.16666666666666
2021/12/16 21:10:41 Rebalance progress: 100
2021/12/16 21:10:41 Failing over: [127.0.0.1:19003]
2021/12/16 21:10:51 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 21:11:51 Removing node(s): [] from the cluster
2021/12/16 21:11:56 Rebalance progress: 29.16666666666666
2021/12/16 21:12:01 Rebalance progress: 29.16666666666666
2021/12/16 21:12:06 Rebalance progress: 100
2021/12/16 21:12:11 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/16 21:12:11 App: TestUndeployWithKVFailover got un-deployed
2021/12/16 21:12:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:12:16 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/16 21:12:16 App: TestUndeployWithKVFailover got un-deployed
2021/12/16 21:12:16 No eventing-con process running
2021/12/16 21:12:16 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/16 21:12:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:12:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 21:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:12:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:12:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 21:12:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 21:12:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (200.22s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/16 21:12:54 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 21:12:54 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/16 21:12:54 No eventing-con process running
2021/12/16 21:12:54 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/16 21:12:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:13:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:13:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:13:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:13:07 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:13:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/16 21:13:13 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 21:13:18 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 21:13:24 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 21:13:29 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 21:13:29 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/16 21:13:29 Starting bucket ops to source bucket
2021/12/16 21:13:49 Waiting for dst bucket item count to get to: 20000 curr count: 12609
2021/12/16 21:13:54 Waiting for dst bucket item count to get to: 20000 curr count: 16975
2021/12/16 21:13:54 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/16 21:13:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:13:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:14:07 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/16 21:14:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:14:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:14:19 Waiting for dst bucket item count to get to: 0 curr count: 18653
2021/12/16 21:14:24 Waiting for dst bucket item count to get to: 0 curr count: 8653
2021/12/16 21:14:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:14:24 Starting bucket ops to source bucket
2021/12/16 21:14:36 Waiting for dst bucket item count to get to: 10000 curr count: 2186
2021/12/16 21:14:41 Waiting for dst bucket item count to get to: 10000 curr count: 5756
2021/12/16 21:14:46 Waiting for dst bucket item count to get to: 10000 curr count: 8570
2021/12/16 21:14:47 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 21:14:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:14:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:14:52 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 21:14:57 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 21:15:02 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 21:15:07 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 21:15:07 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/16 21:15:07 No eventing-con process running
2021/12/16 21:15:07 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/16 21:15:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:15:18 Waiting for dst bucket item count to get to: 0 curr count: 7101
2021/12/16 21:15:23 Waiting for dst bucket item count to get to: 0 curr count: 2101
2021/12/16 21:15:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:15:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:15:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 21:15:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (170.62s)
=== RUN   TestTimerOverwrite
2021/12/16 21:15:45 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 21:15:45 App: TestTimerOverwrite got un-deployed
2021/12/16 21:15:45 No eventing-con process running
2021/12/16 21:15:45 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/16 21:15:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:15:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:15:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:15:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:15:58 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:15:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/16 21:16:04 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 21:16:09 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 21:16:14 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 21:16:19 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 21:16:19 App: TestTimerOverwrite got deployed
2021/12/16 21:16:29 Starting bucket ops to source bucket
2021/12/16 21:16:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:16:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:16:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:16:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:16:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:17:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:17:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:17:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:17:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:17:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:17:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:17:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:17:38 Waiting for dst bucket item count to get to: 5000 curr count: 2509
2021/12/16 21:17:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 21:17:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:17:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:17:54 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 21:17:59 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 21:18:04 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 21:18:09 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 21:18:09 App: TestTimerOverwrite got un-deployed
2021/12/16 21:18:09 No eventing-con process running
2021/12/16 21:18:09 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/16 21:18:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:18:20 Waiting for dst bucket item count to get to: 0 curr count: 2940
2021/12/16 21:18:25 Waiting for dst bucket item count to get to: 0 curr count: 2940
2021/12/16 21:18:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:18:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:18:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 21:18:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (182.00s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/16 21:18:42 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 21:18:42 Removing node(s): [] from the cluster
2021/12/16 21:18:47 Rebalance progress: 100
2021/12/16 21:18:47 Failing over: [127.0.0.1:19003]
2021/12/16 21:18:52 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/16 21:18:52 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/16 21:18:52 No eventing-con process running
2021/12/16 21:18:52 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/16 21:18:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:18:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:19:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:19:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:19:05 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:19:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 21:19:06 Starting bucket ops to source bucket
2021/12/16 21:19:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:19:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:19:16 Context: Post to set retry counter, response code: 200 dump: 
2021/12/16 21:19:16 set retry counter, response &{[] }
2021/12/16 21:20:21 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/16 21:20:21 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/16 21:20:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:20:21 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/16 21:20:26 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/16 21:20:26 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/16 21:20:26 No eventing-con process running
2021/12/16 21:20:26 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/16 21:20:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:20:38 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 21:20:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:20:39 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:20:49 Waiting for dst bucket item count to get to: 0 curr count: 861
2021/12/16 21:20:54 Waiting for dst bucket item count to get to: 0 curr count: 861
2021/12/16 21:20:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:20:54 Removing node(s): [] from the cluster
2021/12/16 21:20:59 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (142.75s)
=== RUN   TestOnUpdateSrcMutation
2021/12/16 21:21:09 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 21:21:09 App: TestOnUpdateSrcMutation got un-deployed
2021/12/16 21:21:09 No eventing-con process running
2021/12/16 21:21:09 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/16 21:21:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:21:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:21:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:21:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:21:23 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:21:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/16 21:21:24 Starting bucket ops to source bucket
2021/12/16 21:21:51 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 21:21:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:21:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:21:56 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 21:22:01 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 21:22:06 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 21:22:11 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 21:22:11 App: TestOnUpdateSrcMutation got un-deployed
2021/12/16 21:22:11 No eventing-con process running
2021/12/16 21:22:11 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/16 21:22:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:22:22 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 21:22:27 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 21:22:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:22:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:22:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.45s)
=== RUN   TestOnDeleteSrcMutation
2021/12/16 21:22:44 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 21:22:44 App: TestOnDeleteSrcMutation got un-deployed
2021/12/16 21:22:44 No eventing-con process running
2021/12/16 21:22:44 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/16 21:22:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:22:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:22:52 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:22:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:22:57 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:22:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/16 21:22:58 Starting bucket ops to source bucket
2021/12/16 21:23:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 21:23:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:23:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:23:32 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 21:23:37 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 21:23:42 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 21:23:47 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 21:23:47 App: TestOnDeleteSrcMutation got un-deployed
2021/12/16 21:23:47 No eventing-con process running
2021/12/16 21:23:47 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/16 21:23:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:23:58 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 21:24:03 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 21:24:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:24:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:24:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.25s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/16 21:24:20 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 21:24:20 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/16 21:24:20 No eventing-con process running
2021/12/16 21:24:20 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/16 21:24:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:24:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:24:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:24:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:24:33 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:24:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 21:24:34 Starting bucket ops to source bucket
2021/12/16 21:25:17 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 21:25:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:25:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:25:22 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 21:25:27 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 21:25:32 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 21:25:37 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 21:25:37 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/16 21:25:37 No eventing-con process running
2021/12/16 21:25:37 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/16 21:25:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:25:48 Waiting for dst bucket item count to get to: 0 curr count: 7280
2021/12/16 21:25:53 Waiting for dst bucket item count to get to: 0 curr count: 2280
2021/12/16 21:25:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:25:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:26:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (109.59s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/16 21:26:10 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 21:26:10 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/16 21:26:10 No eventing-con process running
2021/12/16 21:26:10 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/16 21:26:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:26:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:26:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:26:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:26:23 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:26:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 21:26:24 Starting bucket ops to source bucket
2021/12/16 21:27:08 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 21:27:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:27:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:27:13 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 21:27:18 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 21:27:23 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 21:27:28 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 21:27:28 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/16 21:27:28 No eventing-con process running
2021/12/16 21:27:28 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/16 21:27:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:27:39 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 21:27:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 21:27:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:27:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:27:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (111.48s)
=== RUN   TestInterHandlerRecursion
2021/12/16 21:28:01 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 21:28:01 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 21:28:01 No eventing-con process running
2021/12/16 21:28:01 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/16 21:28:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:28:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:28:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:28:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:28:19 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 21:28:19 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 21:28:19 No eventing-con process running
2021/12/16 21:28:19 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/16 21:28:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:28:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:28:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:28:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:28:32 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:28:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/16 21:28:34 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/16 21:28:39 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 21:28:44 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 21:28:49 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 21:28:54 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 21:28:54 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/16 21:28:54 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:28:54 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/16 21:28:54 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/16 21:28:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:28:59 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 21:29:04 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 21:29:09 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 21:29:14 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 21:29:14 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 21:29:14 No eventing-con process running
2021/12/16 21:29:14 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/16 21:29:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:29:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:29:22 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:29:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:29:32 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 21:29:32 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 21:29:32 No eventing-con process running
2021/12/16 21:29:32 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/16 21:29:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:29:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:29:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:29:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.78s)
=== RUN   TestInterBucketRecursion
2021/12/16 21:29:50 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:29:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 21:29:51 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 21:29:51 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:29:51 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/16 21:29:56 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 21:30:01 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 21:30:06 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 21:30:11 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 21:30:11 App: TestInterBucketRecursion_function1 got deployed
2021/12/16 21:30:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:30:16 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 21:30:21 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 21:30:26 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 21:30:31 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 21:30:31 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/16 21:30:31 No eventing-con process running
2021/12/16 21:30:32 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/16 21:30:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:30:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:30:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:30:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:30:50 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/16 21:30:50 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/16 21:30:50 No eventing-con process running
2021/12/16 21:30:50 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/16 21:30:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:30:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:30:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:31:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.63s)
=== RUN   TestLargeHandler
2021/12/16 21:31:03 Changed force_compress value to false
2021/12/16 21:31:03 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:31:03 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/16 21:31:03 Changed force_compress value to true
2021/12/16 21:31:03 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:31:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/16 21:31:10 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 21:31:15 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 21:31:20 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 21:31:25 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 21:31:25 App: TestLargeHandler_function got deployed
2021/12/16 21:31:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:31:30 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 21:31:35 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 21:31:40 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 21:31:45 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 21:31:45 App: TestLargeHandler_function got un-deployed
2021/12/16 21:31:45 No eventing-con process running
2021/12/16 21:31:45 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/16 21:31:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:31:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:31:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:31:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.41s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/16 21:32:08 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 21:32:08 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 21:32:08 No eventing-con process running
2021/12/16 21:32:08 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/16 21:32:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:32:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:32:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:32:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:32:26 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 21:32:26 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 21:32:26 No eventing-con process running
2021/12/16 21:32:26 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/16 21:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:32:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:32:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:32:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:32:39 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:32:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/16 21:32:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/16 21:32:45 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 21:32:51 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 21:32:56 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 21:33:01 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 21:33:01 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/16 21:33:01 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:33:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/16 21:33:02 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/16 21:33:07 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 21:33:12 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 21:33:17 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 21:33:22 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 21:33:22 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/16 21:33:22 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/16 21:33:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:33:27 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 21:33:32 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 21:33:37 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 21:33:42 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 21:33:42 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 21:33:42 No eventing-con process running
2021/12/16 21:33:42 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/16 21:33:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:33:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:33:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:33:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:33:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:34:01 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 21:34:06 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 21:34:11 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 21:34:16 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 21:34:16 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 21:34:16 No eventing-con process running
2021/12/16 21:34:16 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/16 21:34:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:34:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:34:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:34:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.40s)
=== RUN   TestAllowInterBucketRecursion
2021/12/16 21:34:34 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:34:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 21:34:36 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 21:34:41 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 21:34:46 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 21:34:51 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 21:34:56 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 21:34:56 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/16 21:34:56 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:34:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/16 21:34:57 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/16 21:35:02 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 21:35:07 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 21:35:12 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 21:35:17 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 21:35:17 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/16 21:35:17 TestAllowInterBucketRecursion_function2 is deployed
2021/12/16 21:35:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:35:22 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 21:35:27 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 21:35:32 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 21:35:37 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 21:35:37 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/16 21:35:37 No eventing-con process running
2021/12/16 21:35:37 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/16 21:35:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:35:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:35:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:35:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:35:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:35:56 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 21:36:01 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 21:36:06 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 21:36:11 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 21:36:11 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/16 21:36:11 No eventing-con process running
2021/12/16 21:36:11 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/16 21:36:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:36:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:36:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:36:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (114.69s)
=== RUN   TestInterBucketRecursion2
2021/12/16 21:36:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:36:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:36:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:36:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:36:54 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:36:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 21:37:00 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 21:37:05 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 21:37:10 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 21:37:15 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 21:37:15 App: TestInterBucketRecursion2_function1 got deployed
2021/12/16 21:37:15 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:37:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/16 21:37:21 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 21:37:26 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 21:37:31 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 21:37:36 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 21:37:36 App: TestInterBucketRecursion2_function2 got deployed
2021/12/16 21:37:36 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:37:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/16 21:37:42 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 21:37:47 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 21:37:52 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 21:37:57 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 21:37:57 App: TestInterBucketRecursion2_function3 got deployed
2021/12/16 21:37:57 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:37:57 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/16 21:37:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:38:02 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 21:38:07 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 21:38:12 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 21:38:17 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 21:38:17 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/16 21:38:17 No eventing-con process running
2021/12/16 21:38:18 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/16 21:38:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:38:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:38:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:38:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:38:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:38:36 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 21:38:41 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 21:38:46 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 21:38:51 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 21:38:51 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/16 21:38:51 No eventing-con process running
2021/12/16 21:38:51 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/16 21:38:52 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:38:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:38:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:39:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:39:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:39:09 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 21:39:14 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 21:39:19 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 21:39:24 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 21:39:24 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/16 21:39:24 No eventing-con process running
2021/12/16 21:39:24 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/16 21:39:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:39:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:39:32 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:39:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:39:42 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/16 21:39:42 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/16 21:39:42 No eventing-con process running
2021/12/16 21:39:42 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/16 21:39:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:39:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:39:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:39:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.45s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/16 21:40:03 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 21:40:08 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/16 21:40:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:40:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:40:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:40:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:41:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:41:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:41:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:41:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:42:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:42:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:42:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:42:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:43:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:43:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:43:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:43:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:44:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:44:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:44:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:44:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:45:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:45:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:45:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 21:45:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/16 21:46:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:46:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 21:46:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 21:46:16 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/16 21:46:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:46:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 21:46:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 21:46:22 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/16 21:46:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:46:27 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/16 21:46:27 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/16 21:46:27 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:46:27 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/16 21:46:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:46:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:46:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:46:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:46:45 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:46:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 21:46:51 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 21:46:56 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 21:47:01 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 21:47:06 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 21:47:06 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 06dee0ebd7f756acf894cdd964b016a4
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 072fe370342ec7975c4d2c9c9e297d1d
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 40a17e08bf3c4769379a67768846dd4e
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/16 21:47:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:47:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 21:47:11 Rebalance progress: 29.16666666666666
2021/12/16 21:47:16 Rebalance progress: 29.16666666666666
2021/12/16 21:47:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 06dee0ebd7f756acf894cdd964b016a4
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 072fe370342ec7975c4d2c9c9e297d1d
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 40a17e08bf3c4769379a67768846dd4e
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/16 21:47:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:47:21 handler_test.go:1674: Deleting source bucket: default
2021/12/16 21:47:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 21:47:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 21:47:28 Rebalance progress: 29.16666666666666
2021/12/16 21:47:33 Rebalance progress: 29.16666666666666
2021/12/16 21:47:38 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 900
	vb list dump: [0-14, 16-31, 33-37, 40-44, 47-57, 59-68, 70-86, 88-90, 92, 94, 96-100, 102-131, 133, 136, 138-139, 141-142, 144-163, 165-172, 174, 176-181, 183-189, 191-208, 210-211, 213, 215-227, 229, 232-235, 237-245, 247-266, 268, 270-276, 278, 280-291, 293-298, 300-315, 318-331, 333-347, 349-353, 355-366, 368-371, 373-378, 380-382, 384-397, 399-404, 407-408, 410-414, 416, 418-426, 428-443, 445-511, 514, 517-518, 520, 522-534, 537-544, 546-547, 549-552, 554-562, 564, 566-568, 570-601, 603, 605-606, 608-611, 613, 615-616, 618-625, 627, 629-661, 663-664, 666-674, 677-681, 683-684, 686, 688-691, 693, 695-700, 702-710, 712-715, 717-724, 726-730, 732-760, 762-768, 770-772, 774, 776-785, 787-788, 790-800, 802-808, 810-817, 820-830, 832-834, 836-847, 849-858, 860-871, 873-880, 883-884, 886-905, 907-911, 913-929, 931-934, 936-940, 942-946, 948-955, 957-959, 961-966, 968-974, 976-977, 979, 981-989, 991-1007, 1009-1016, 1018-1023]
	stream status: stopped
	len: 124
	vb list dump: [15, 32, 38-39, 45-46, 58, 69, 87, 91, 93, 95, 101, 132, 134-135, 137, 140, 143, 164, 173, 175, 182, 190, 209, 212, 214, 228, 230-231, 236, 246, 267, 269, 277, 279, 292, 299, 316-317, 332, 348, 354, 367, 372, 379, 383, 398, 405-406, 409, 415, 417, 427, 444, 512-513, 515-516, 519, 521, 535-536, 545, 548, 553, 563, 565, 569, 602, 604, 607, 612, 614, 617, 626, 628, 662, 665, 675-676, 682, 685, 687, 692, 694, 701, 711, 716, 725, 731, 761, 769, 773, 775, 786, 789, 801, 809, 818-819, 831, 835, 848, 859, 872, 881-882, 885, 906, 912, 930, 935, 941, 947, 956, 960, 967, 975, 978, 980, 990, 1008, 1017]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 06dee0ebd7f756acf894cdd964b016a4
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-14, 16-31, 33-37, 40-44, 47-57, 59-68, 70-86, 88-90, 92, 94, 96-100, 102-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [114-131, 133, 136, 138-139, 141-142, 144-163, 165-172, 174, 176-181, 183-189, 191-208, 210-211, 213, 215-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [229, 232-235, 237-245, 247-266, 268, 270-276, 278, 280-291, 293-298, 300-315, 318-331, 333-341]
Producer node: 172.31.2.134:9300	Node UUID: 072fe370342ec7975c4d2c9c9e297d1d
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 99
	v: [911, 913-929, 931-934, 936-940, 942-946, 948-955, 957-959, 961-966, 968-974, 976-977, 979, 981-989, 991-1007, 1009-1016, 1018-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [683-684, 686, 688-691, 693, 695-700, 702-710, 712-715, 717-724, 726-730, 732-760, 762-768, 770-772, 774, 776-785, 787-788, 790-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-800, 802-808, 810-817, 820-830, 832-834, 836-847, 849-858, 860-871, 873-880, 883-884, 886-905, 907-910]
Producer node: 127.0.0.1:9303	Node UUID: 40a17e08bf3c4769379a67768846dd4e
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [342-347, 349-353, 355-366, 368-371, 373-378, 380-382, 384-397, 399-404, 407-408, 410-414, 416, 418-426, 428-443, 445-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [456-511, 514, 517-518, 520, 522-534, 537-544, 546-547, 549-552, 554-562, 564, 566-568]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570-601, 603, 605-606, 608-611, 613, 615-616, 618-625, 627, 629-661, 663-664, 666-674, 677-681]


2021/12/16 21:47:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:47:44 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/16 21:47:44 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/16 21:47:49 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/16 21:47:49 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/16 21:47:49 eventingSetup.go:46: No eventing-con process running
2021/12/16 21:47:49 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/16 21:47:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:47:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:47:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:48:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:48:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/16 21:48:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 21:48:07 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (492.05s)
=== RUN   TestN1QLRecursion
2021/12/16 21:48:10 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 21:48:10 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/16 21:48:10 Removing node(s): [] from the cluster
2021/12/16 21:48:15 Rebalance progress: 100
2021/12/16 21:48:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:48:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:48:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:48:38 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:48:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 21:48:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 21:48:45 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 21:48:50 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 21:48:55 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 21:49:00 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 21:49:00 App: TestN1QLRecursion_function1 got deployed
2021/12/16 21:49:00 TestN1QLRecursion_function1 is deployed
2021/12/16 21:49:00 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:49:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/16 21:49:01 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/16 21:49:06 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 21:49:11 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 21:49:16 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 21:49:21 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 21:49:21 App: TestN1QLRecursion_function3 got deployed
2021/12/16 21:49:21 TestN1QLRecursion_function3 is deployed
2021/12/16 21:49:21 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:49:21 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/16 21:49:21 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/16 21:49:21 Success: TestN1QLRecursion_function2 is not deployed
2021/12/16 21:49:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:49:26 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 21:49:31 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 21:49:36 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 21:49:41 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 21:49:41 App: TestN1QLRecursion_function1 got un-deployed
2021/12/16 21:49:41 No eventing-con process running
2021/12/16 21:49:41 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/16 21:49:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:49:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:49:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:49:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:49:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:50:00 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 21:50:05 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 21:50:10 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 21:50:15 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 21:50:15 App: TestN1QLRecursion_function3 got un-deployed
2021/12/16 21:50:15 No eventing-con process running
2021/12/16 21:50:15 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/16 21:50:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:50:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:50:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:50:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:50:30 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 21:50:35 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.80s)
=== RUN   TestN1QLAllowRecursion
2021/12/16 21:50:35 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 21:50:36 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/16 21:50:36 Removing node(s): [] from the cluster
2021/12/16 21:50:41 Rebalance progress: 100
2021/12/16 21:50:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:50:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:50:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:51:04 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:51:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 21:51:05 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 21:51:10 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 21:51:15 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 21:51:20 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 21:51:25 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 21:51:25 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/16 21:51:25 TestN1QLAllowRecursion_function1 is deployed
2021/12/16 21:51:25 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:51:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/16 21:51:26 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/16 21:51:31 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 21:51:36 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 21:51:41 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 21:51:46 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 21:51:46 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/16 21:51:46 TestN1QLAllowRecursion_function3 is deployed
2021/12/16 21:51:46 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:51:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/16 21:51:47 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/16 21:51:52 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 21:51:57 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 21:52:02 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 21:52:07 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 21:52:07 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/16 21:52:07 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/16 21:52:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:52:12 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 21:52:17 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 21:52:22 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 21:52:27 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 21:52:27 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/16 21:52:27 No eventing-con process running
2021/12/16 21:52:28 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/16 21:52:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:52:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:52:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:52:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:52:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:52:46 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 21:52:51 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 21:52:56 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 21:53:01 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 21:53:01 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/16 21:53:01 No eventing-con process running
2021/12/16 21:53:01 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/16 21:53:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:53:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:53:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:53:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:53:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:53:20 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 21:53:25 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 21:53:30 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 21:53:35 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 21:53:35 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/16 21:53:35 No eventing-con process running
2021/12/16 21:53:35 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/16 21:53:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:53:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:53:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:53:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:53:51 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 21:53:56 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.23s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/16 21:53:56 Sleeping for some time
2021/12/16 21:54:06 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 21:54:06 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/16 21:54:06 No eventing-con process running
2021/12/16 21:54:06 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/16 21:54:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:54:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:54:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:54:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:54:19 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:54:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 21:54:21 Starting bucket ops to source bucket
2021/12/16 21:55:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:55:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 21:55:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 21:55:15 Starting bucket ops to source bucket
2021/12/16 21:55:16 Starting bucket ops to source bucket
2021/12/16 21:56:22 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/16 21:56:22 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/16 21:56:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:56:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:56:27 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 21:56:32 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 21:56:37 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 21:56:42 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 21:56:42 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/16 21:56:42 No eventing-con process running
2021/12/16 21:56:42 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/16 21:56:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:56:54 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 21:56:59 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/16 21:56:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:57:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:57:11 Waiting for dst bucket item count to get to: 0 curr count: 5927
2021/12/16 21:57:16 Waiting for dst bucket item count to get to: 0 curr count: 5927
2021/12/16 21:57:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.94s)
=== RUN   TestTimerOverWriteSameReference
2021/12/16 21:57:21 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 21:57:21 App: TestTimerOverWriteSameReference got un-deployed
2021/12/16 21:57:21 No eventing-con process running
2021/12/16 21:57:21 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/16 21:57:22 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:57:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:57:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:57:34 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/16 21:57:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/16 21:57:40 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 21:57:45 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 21:57:50 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 21:57:55 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 21:57:55 App: TestTimerOverWriteSameReference got deployed
2021/12/16 21:58:05 Starting bucket ops to source bucket
2021/12/16 21:59:07 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/16 21:59:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 21:59:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 21:59:22 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 21:59:27 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 21:59:32 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 21:59:37 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 21:59:37 App: TestTimerOverWriteSameReference got un-deployed
2021/12/16 21:59:37 No eventing-con process running
2021/12/16 21:59:37 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/16 21:59:39 Context: Bucket flush, response code: 200 dump: 
2021/12/16 21:59:49 Waiting for dst bucket item count to get to: 0 curr count: 921
2021/12/16 21:59:54 Waiting for dst bucket item count to get to: 0 curr count: 421
2021/12/16 21:59:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 21:59:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:00:06 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 22:00:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.91s)
=== RUN   TestCancelTimerBucketop
2021/12/16 22:00:11 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 22:00:11 App: TestCancelTimerBucketop got un-deployed
2021/12/16 22:00:11 No eventing-con process running
2021/12/16 22:00:11 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/16 22:00:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:00:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:00:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:00:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:00:24 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:00:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/16 22:00:30 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 22:00:35 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 22:00:40 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 22:00:45 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 22:00:45 App: TestCancelTimerBucketop got deployed
2021/12/16 22:00:45 Starting bucket ops to source bucket
2021/12/16 22:00:53 Starting bucket ops to source bucket
2021/12/16 22:01:55 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 22:01:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:01:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:02:00 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 22:02:05 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 22:02:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 22:02:15 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 22:02:15 App: TestCancelTimerBucketop got un-deployed
2021/12/16 22:02:15 No eventing-con process running
2021/12/16 22:02:15 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/16 22:02:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:02:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 22:02:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:02:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:02:39 Waiting for dst bucket item count to get to: 0 curr count: 972
2021/12/16 22:02:44 Waiting for dst bucket item count to get to: 0 curr count: 472
2021/12/16 22:02:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.29s)
=== RUN   TestJSExpiryDate
2021/12/16 22:02:49 Starting bucket ops to source bucket
2021/12/16 22:03:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 22:03:07 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/16 22:03:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/16 22:03:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:03:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:03:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:03:38 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 22:03:43 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 22:03:48 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 22:03:53 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 22:03:53 App: TestJSExpiryDate got un-deployed
2021/12/16 22:03:53 No eventing-con process running
2021/12/16 22:03:53 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/16 22:03:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:04:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:04:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:04:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.68s)
=== RUN   TestBinaryDoc
2021/12/16 22:04:12 Starting bucket ops to source bucket
2021/12/16 22:04:13 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:04:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/16 22:04:14 Starting bucket ops to source bucket
2021/12/16 22:04:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:04:14 Deleting source bucket docs now
2021/12/16 22:04:14 Starting bucket ops to source bucket
2021/12/16 22:04:51 Waiting for dst bucket item count to get to: 32 curr count: 24
2021/12/16 22:04:56 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/16 22:04:56 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/16 22:04:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:04:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:05:01 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 22:05:06 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 22:05:11 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 22:05:16 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 22:05:16 App: TestBinaryDoc got un-deployed
2021/12/16 22:05:16 No eventing-con process running
2021/12/16 22:05:16 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/16 22:05:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:05:28 Waiting for dst bucket item count to get to: 0 curr count: 22
2021/12/16 22:05:33 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/16 22:05:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:05:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:05:45 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/16 22:05:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.32s)
=== RUN   TestConstantBindings
2021/12/16 22:05:45 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:05:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/16 22:05:46 Starting bucket ops to source bucket
2021/12/16 22:05:52 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/16 22:05:57 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/16 22:06:02 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/16 22:06:07 Waiting for dst bucket item count to get to: 6000 curr count: 2196
2021/12/16 22:06:12 Waiting for dst bucket item count to get to: 6000 curr count: 2196
2021/12/16 22:06:12 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/16 22:06:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:06:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:06:17 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 22:06:22 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 22:06:27 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 22:06:32 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 22:06:32 App: TestConstantBindings got un-deployed
2021/12/16 22:06:32 No eventing-con process running
2021/12/16 22:06:32 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/16 22:06:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:06:44 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 22:06:49 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 22:06:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:06:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:07:01 Waiting for dst bucket item count to get to: 0 curr count: 5020
2021/12/16 22:07:06 Waiting for dst bucket item count to get to: 0 curr count: 5020
2021/12/16 22:07:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.78s)
=== RUN   TestCodeInsight
2021/12/16 22:07:11 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 22:07:11 App: TestCodeInsight got un-deployed
2021/12/16 22:07:11 No eventing-con process running
2021/12/16 22:07:11 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/16 22:07:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:07:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:07:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:07:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:07:24 Starting bucket ops to source bucket
2021/12/16 22:07:25 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:07:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/16 22:07:31 Waiting for app: TestCodeInsight to get deployed
2021/12/16 22:07:36 Waiting for app: TestCodeInsight to get deployed
2021/12/16 22:07:41 Waiting for app: TestCodeInsight to get deployed
2021/12/16 22:07:46 Waiting for app: TestCodeInsight to get deployed
2021/12/16 22:07:46 App: TestCodeInsight got deployed
2021/12/16 22:07:51 Waiting for dst bucket item count to get to: 1000 curr count: 439
2021/12/16 22:07:51 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 22:07:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:07:56 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 22:08:01 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 22:08:06 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 22:08:11 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 22:08:11 App: TestCodeInsight got un-deployed
2021/12/16 22:08:11 No eventing-con process running
2021/12/16 22:08:11 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/16 22:08:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:08:23 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 22:08:28 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 22:08:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:08:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:08:40 Waiting for dst bucket item count to get to: 0 curr count: 567
2021/12/16 22:08:45 Waiting for dst bucket item count to get to: 0 curr count: 567
2021/12/16 22:08:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.28s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.07s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13020.143s

Functional tests took 3 hours, 37 minutes and 4 seconds.

N1ql tests

2021/12/16 22:08:56 Started n1ql
2021/12/16 22:09:05 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/16 22:09:06 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/16 22:09:06 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 22:09:06 Removing node(s): [] from the cluster
2021/12/16 22:09:11 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/16 22:09:16 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/16 22:09:16 App: TestRecursiveMutationN1QL got un-deployed
2021/12/16 22:09:16 No eventing-con process running
2021/12/16 22:09:16 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/16 22:09:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:09:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:09:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:09:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:09:30 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:09:30 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.56s)
=== RUN   TestFlexReset1
2021/12/16 22:09:35 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 22:09:35 App: TestFlexReset1 got un-deployed
2021/12/16 22:09:35 No eventing-con process running
2021/12/16 22:09:35 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/16 22:09:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:09:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:09:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:09:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:09:48 Starting bucket ops to source bucket
2021/12/16 22:09:49 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:09:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/16 22:09:55 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 22:10:00 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 22:10:05 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 22:10:10 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 22:10:10 App: TestFlexReset1 got deployed
2021/12/16 22:10:15 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/16 22:10:15 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 22:10:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:10:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:10:20 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 22:10:25 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 22:10:30 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 22:10:35 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 22:10:35 App: TestFlexReset1 got un-deployed
2021/12/16 22:10:35 No eventing-con process running
2021/12/16 22:10:35 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/16 22:10:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:10:47 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 22:10:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:10:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:10:59 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 22:11:04 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 22:11:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.17s)
=== RUN   TestFlexReset2
2021/12/16 22:11:09 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 22:11:09 App: TestFlexReset2 got un-deployed
2021/12/16 22:11:09 No eventing-con process running
2021/12/16 22:11:09 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/16 22:11:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:11:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:11:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:11:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:11:23 Starting bucket ops to source bucket
2021/12/16 22:11:23 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:11:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/16 22:11:29 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 22:11:34 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 22:11:39 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 22:11:44 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 22:11:44 App: TestFlexReset2 got deployed
2021/12/16 22:11:49 Waiting for dst bucket item count to get to: 1000 curr count: 550
2021/12/16 22:11:49 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 22:11:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:11:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:11:55 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 22:12:00 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 22:12:05 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 22:12:10 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 22:12:10 App: TestFlexReset2 got un-deployed
2021/12/16 22:12:10 No eventing-con process running
2021/12/16 22:12:10 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/16 22:12:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:12:21 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 22:12:26 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 22:12:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:12:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:12:38 Waiting for dst bucket item count to get to: 0 curr count: 780
2021/12/16 22:12:43 Waiting for dst bucket item count to get to: 0 curr count: 280
2021/12/16 22:12:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.26s)
=== RUN   TestN1QLLabelledBreak
2021/12/16 22:12:48 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 22:12:48 App: TestN1QLLabelledBreak got un-deployed
2021/12/16 22:12:48 No eventing-con process running
2021/12/16 22:12:48 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/16 22:12:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:12:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:12:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:13:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:13:02 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:13:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/16 22:13:08 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 22:13:13 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 22:13:18 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 22:13:23 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 22:13:23 App: TestN1QLLabelledBreak got deployed
2021/12/16 22:13:23 Starting bucket ops to source bucket
2021/12/16 22:13:32 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/16 22:13:37 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/16 22:13:42 Waiting for dst bucket item count to get to: 10000 curr count: 941
2021/12/16 22:13:47 Waiting for dst bucket item count to get to: 10000 curr count: 2927
2021/12/16 22:13:52 Waiting for dst bucket item count to get to: 10000 curr count: 5018
2021/12/16 22:13:57 Waiting for dst bucket item count to get to: 10000 curr count: 7083
2021/12/16 22:14:02 Waiting for dst bucket item count to get to: 10000 curr count: 9107
2021/12/16 22:14:08 Waiting for dst bucket item count to get to: 10000 curr count: 9755
2021/12/16 22:14:08 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 22:14:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:14:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:14:13 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 22:14:18 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 22:14:23 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 22:14:28 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 22:14:28 App: TestN1QLLabelledBreak got un-deployed
2021/12/16 22:14:28 No eventing-con process running
2021/12/16 22:14:28 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/16 22:14:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:14:40 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 22:14:45 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 22:14:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:14:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:14:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 22:14:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (133.17s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/16 22:15:01 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 22:15:01 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/16 22:15:01 No eventing-con process running
2021/12/16 22:15:01 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/16 22:15:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:15:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:15:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:15:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:15:15 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:15:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/16 22:15:21 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 22:15:26 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 22:15:31 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 22:15:36 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 22:15:36 App: TestN1QLUnlabelledBreak got deployed
2021/12/16 22:15:36 Starting bucket ops to source bucket
2021/12/16 22:15:46 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/16 22:15:51 Waiting for dst bucket item count to get to: 10000 curr count: 406
2021/12/16 22:15:56 Waiting for dst bucket item count to get to: 10000 curr count: 406
2021/12/16 22:16:01 Waiting for dst bucket item count to get to: 10000 curr count: 5379
2021/12/16 22:16:06 Waiting for dst bucket item count to get to: 10000 curr count: 5379
2021/12/16 22:16:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 22:16:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:16:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:16:11 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 22:16:16 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 22:16:21 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 22:16:26 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 22:16:26 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/16 22:16:26 No eventing-con process running
2021/12/16 22:16:26 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/16 22:16:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:16:38 Waiting for dst bucket item count to get to: 0 curr count: 2842
2021/12/16 22:16:43 Waiting for dst bucket item count to get to: 0 curr count: 342
2021/12/16 22:16:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:16:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:16:55 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 22:16:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.12s)
=== RUN   TestN1QLThrowStatement
2021/12/16 22:17:00 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 22:17:00 App: TestN1QLThrowStatement got un-deployed
2021/12/16 22:17:00 No eventing-con process running
2021/12/16 22:17:00 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/16 22:17:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:17:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:17:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:17:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:17:13 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:17:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/16 22:17:19 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 22:17:24 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 22:17:29 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 22:17:34 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 22:17:34 App: TestN1QLThrowStatement got deployed
2021/12/16 22:17:34 Starting bucket ops to source bucket
2021/12/16 22:17:43 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/16 22:17:48 Waiting for dst bucket item count to get to: 10000 curr count: 196
2021/12/16 22:17:53 Waiting for dst bucket item count to get to: 10000 curr count: 650
2021/12/16 22:17:58 Waiting for dst bucket item count to get to: 10000 curr count: 2514
2021/12/16 22:18:03 Waiting for dst bucket item count to get to: 10000 curr count: 4564
2021/12/16 22:18:08 Waiting for dst bucket item count to get to: 10000 curr count: 6620
2021/12/16 22:18:13 Waiting for dst bucket item count to get to: 10000 curr count: 8643
2021/12/16 22:18:18 Waiting for dst bucket item count to get to: 10000 curr count: 9527
2021/12/16 22:18:19 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 22:18:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:18:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:18:24 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 22:18:29 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 22:18:34 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 22:18:39 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 22:18:39 App: TestN1QLThrowStatement got un-deployed
2021/12/16 22:18:39 No eventing-con process running
2021/12/16 22:18:39 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/16 22:18:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:18:51 Waiting for dst bucket item count to get to: 0 curr count: 3950
2021/12/16 22:18:56 Waiting for dst bucket item count to get to: 0 curr count: 3950
2021/12/16 22:18:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:18:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:19:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 22:19:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (132.82s)
=== RUN   TestN1QLNestedForLoop
2021/12/16 22:19:12 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 22:19:12 App: TestN1QLNestedForLoop got un-deployed
2021/12/16 22:19:12 No eventing-con process running
2021/12/16 22:19:12 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/16 22:19:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:19:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:19:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:19:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:19:26 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:19:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/16 22:19:32 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 22:19:37 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 22:19:42 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 22:19:47 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 22:19:47 App: TestN1QLNestedForLoop got deployed
2021/12/16 22:19:47 Starting bucket ops to source bucket
2021/12/16 22:19:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 22:20:02 Waiting for dst bucket item count to get to: 5000 curr count: 67
2021/12/16 22:20:07 Waiting for dst bucket item count to get to: 5000 curr count: 67
2021/12/16 22:20:12 Waiting for dst bucket item count to get to: 5000 curr count: 1909
2021/12/16 22:20:17 Waiting for dst bucket item count to get to: 5000 curr count: 1909
2021/12/16 22:20:22 Waiting for dst bucket item count to get to: 5000 curr count: 4297
2021/12/16 22:20:27 Waiting for dst bucket item count to get to: 5000 curr count: 4297
2021/12/16 22:20:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 22:20:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:20:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:20:32 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 22:20:37 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 22:20:42 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 22:20:47 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 22:20:47 App: TestN1QLNestedForLoop got un-deployed
2021/12/16 22:20:47 No eventing-con process running
2021/12/16 22:20:47 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/16 22:20:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:20:59 Waiting for dst bucket item count to get to: 0 curr count: 4492
2021/12/16 22:21:04 Waiting for dst bucket item count to get to: 0 curr count: 1992
2021/12/16 22:21:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:21:06 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:21:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 22:21:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.16s)
=== RUN   TestN1QLAutoClose
2021/12/16 22:21:21 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 22:21:21 App: TestN1QLAutoClose got un-deployed
2021/12/16 22:21:21 No eventing-con process running
2021/12/16 22:21:21 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/16 22:21:22 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:21:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:21:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:21:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:21:34 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:21:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/16 22:21:40 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 22:21:45 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 22:21:50 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 22:21:55 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 22:21:55 App: TestN1QLAutoClose got deployed
2021/12/16 22:21:55 Starting bucket ops to source bucket
2021/12/16 22:22:01 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/16 22:22:06 Waiting for dst bucket item count to get to: 500 curr count: 5
2021/12/16 22:22:11 Waiting for dst bucket item count to get to: 500 curr count: 5
2021/12/16 22:22:16 Waiting for dst bucket item count to get to: 500 curr count: 25
2021/12/16 22:22:21 Waiting for dst bucket item count to get to: 500 curr count: 25
2021/12/16 22:22:26 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/16 22:22:31 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/16 22:22:36 Waiting for dst bucket item count to get to: 500 curr count: 67
2021/12/16 22:22:41 Waiting for dst bucket item count to get to: 500 curr count: 67
2021/12/16 22:22:46 Waiting for dst bucket item count to get to: 500 curr count: 89
2021/12/16 22:22:51 Waiting for dst bucket item count to get to: 500 curr count: 89
2021/12/16 22:22:56 Waiting for dst bucket item count to get to: 500 curr count: 109
2021/12/16 22:23:01 Waiting for dst bucket item count to get to: 500 curr count: 109
2021/12/16 22:23:06 Waiting for dst bucket item count to get to: 500 curr count: 109
2021/12/16 22:23:11 Waiting for dst bucket item count to get to: 500 curr count: 130
2021/12/16 22:23:16 Waiting for dst bucket item count to get to: 500 curr count: 130
2021/12/16 22:23:21 Waiting for dst bucket item count to get to: 500 curr count: 153
2021/12/16 22:23:26 Waiting for dst bucket item count to get to: 500 curr count: 153
2021/12/16 22:23:31 Waiting for dst bucket item count to get to: 500 curr count: 174
2021/12/16 22:23:36 Waiting for dst bucket item count to get to: 500 curr count: 174
2021/12/16 22:23:41 Waiting for dst bucket item count to get to: 500 curr count: 195
2021/12/16 22:23:46 Waiting for dst bucket item count to get to: 500 curr count: 195
2021/12/16 22:23:51 Waiting for dst bucket item count to get to: 500 curr count: 218
2021/12/16 22:23:56 Waiting for dst bucket item count to get to: 500 curr count: 218
2021/12/16 22:24:01 Waiting for dst bucket item count to get to: 500 curr count: 238
2021/12/16 22:24:06 Waiting for dst bucket item count to get to: 500 curr count: 238
2021/12/16 22:24:11 Waiting for dst bucket item count to get to: 500 curr count: 260
2021/12/16 22:24:16 Waiting for dst bucket item count to get to: 500 curr count: 268
2021/12/16 22:24:21 Waiting for dst bucket item count to get to: 500 curr count: 281
2021/12/16 22:24:26 Waiting for dst bucket item count to get to: 500 curr count: 295
2021/12/16 22:24:31 Waiting for dst bucket item count to get to: 500 curr count: 304
2021/12/16 22:24:36 Waiting for dst bucket item count to get to: 500 curr count: 316
2021/12/16 22:24:41 Waiting for dst bucket item count to get to: 500 curr count: 325
2021/12/16 22:24:46 Waiting for dst bucket item count to get to: 500 curr count: 336
2021/12/16 22:24:51 Waiting for dst bucket item count to get to: 500 curr count: 346
2021/12/16 22:24:56 Waiting for dst bucket item count to get to: 500 curr count: 358
2021/12/16 22:25:01 Waiting for dst bucket item count to get to: 500 curr count: 367
2021/12/16 22:25:06 Waiting for dst bucket item count to get to: 500 curr count: 379
2021/12/16 22:25:11 Waiting for dst bucket item count to get to: 500 curr count: 389
2021/12/16 22:25:16 Waiting for dst bucket item count to get to: 500 curr count: 399
2021/12/16 22:25:21 Waiting for dst bucket item count to get to: 500 curr count: 411
2021/12/16 22:25:26 Waiting for dst bucket item count to get to: 500 curr count: 424
2021/12/16 22:25:31 Waiting for dst bucket item count to get to: 500 curr count: 432
2021/12/16 22:25:36 Waiting for dst bucket item count to get to: 500 curr count: 443
2021/12/16 22:25:41 Waiting for dst bucket item count to get to: 500 curr count: 454
2021/12/16 22:25:46 Waiting for dst bucket item count to get to: 500 curr count: 466
2021/12/16 22:25:51 Waiting for dst bucket item count to get to: 500 curr count: 477
2021/12/16 22:25:56 Waiting for dst bucket item count to get to: 500 curr count: 496
2021/12/16 22:26:01 Waiting for dst bucket item count to get to: 500 curr count: 496
2021/12/16 22:26:01 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/16 22:26:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:26:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:26:06 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 22:26:11 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 22:26:16 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 22:26:22 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 22:26:22 App: TestN1QLAutoClose got un-deployed
2021/12/16 22:26:22 No eventing-con process running
2021/12/16 22:26:22 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/16 22:26:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:26:33 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 22:26:38 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/16 22:26:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:26:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:26:50 Waiting for dst bucket item count to get to: 0 curr count: 329
2021/12/16 22:26:55 Waiting for dst bucket item count to get to: 0 curr count: 329
2021/12/16 22:26:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.57s)
=== RUN   TestN1QLPosParams
2021/12/16 22:27:00 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 22:27:00 App: TestN1QLPosParams got un-deployed
2021/12/16 22:27:00 No eventing-con process running
2021/12/16 22:27:00 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/16 22:27:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:27:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:27:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:27:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:27:14 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:27:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/16 22:27:20 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 22:27:25 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 22:27:30 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 22:27:35 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 22:27:35 App: TestN1QLPosParams got deployed
2021/12/16 22:27:35 Starting bucket ops to source bucket
2021/12/16 22:27:45 Waiting for dst bucket item count to get to: 5000 curr count: 369
2021/12/16 22:27:50 Waiting for dst bucket item count to get to: 5000 curr count: 1105
2021/12/16 22:27:50 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 22:27:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:27:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:27:55 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 22:28:00 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 22:28:05 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 22:28:10 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 22:28:10 App: TestN1QLPosParams got un-deployed
2021/12/16 22:28:10 No eventing-con process running
2021/12/16 22:28:10 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/16 22:28:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:28:22 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 22:28:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 22:28:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:28:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:28:39 Waiting for dst bucket item count to get to: 0 curr count: 4318
2021/12/16 22:28:44 Waiting for dst bucket item count to get to: 0 curr count: 1818
2021/12/16 22:28:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.44s)
=== RUN   TestN1QLExhaustConnPool
2021/12/16 22:28:49 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 22:28:49 App: TestN1QLExhaustConnPool got un-deployed
2021/12/16 22:28:49 No eventing-con process running
2021/12/16 22:28:49 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/16 22:28:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:28:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:28:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:29:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:29:02 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:29:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/16 22:29:08 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 22:29:13 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 22:29:18 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 22:29:23 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 22:29:23 App: TestN1QLExhaustConnPool got deployed
2021/12/16 22:29:23 Starting bucket ops to source bucket
2021/12/16 22:29:28 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 22:29:33 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 22:29:33 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 22:29:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:29:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:29:39 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 22:29:44 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 22:29:49 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 22:29:54 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 22:29:54 App: TestN1QLExhaustConnPool got un-deployed
2021/12/16 22:29:54 No eventing-con process running
2021/12/16 22:29:54 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/16 22:29:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:30:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 22:30:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:30:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:30:17 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 22:30:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (93.88s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/16 22:30:22 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 22:30:22 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/16 22:30:22 No eventing-con process running
2021/12/16 22:30:22 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/16 22:30:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:30:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:30:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:30:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:30:36 Starting bucket ops to source bucket
2021/12/16 22:30:54 Starting bucket ops to source bucket
2021/12/16 22:31:11 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 8051
2021/12/16 22:31:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/16 22:31:17 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 22:31:22 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 22:31:27 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 22:31:32 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 22:31:32 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/16 22:31:32 Starting bucket ops to source bucket
2021/12/16 22:31:37 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 22:31:42 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 22:31:47 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 22:31:52 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 22:31:57 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 22:32:02 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 22:32:07 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 22:32:12 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 22:32:17 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 22:32:22 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 22:32:22 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/16 22:32:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:32:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:32:28 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 22:32:33 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 22:32:38 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 22:32:43 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 22:32:43 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/16 22:32:43 No eventing-con process running
2021/12/16 22:32:43 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/16 22:32:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:32:55 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/16 22:32:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:32:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:33:06 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/16 22:33:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (168.90s)
=== RUN   TestN1QLQueryConsistency
2021/12/16 22:33:11 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 22:33:11 App: n1qlquery_consistency got un-deployed
2021/12/16 22:33:11 No eventing-con process running
2021/12/16 22:33:11 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/16 22:33:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:33:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:33:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:33:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:33:25 Starting bucket ops to source bucket
2021/12/16 22:33:28 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:33:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/16 22:33:34 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 22:33:39 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 22:33:44 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 22:33:49 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 22:33:49 App: n1qlquery_consistency got deployed
2021/12/16 22:33:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 22:33:59 Waiting for dst bucket item count to get to: 5000 curr count: 4378
2021/12/16 22:34:04 Waiting for dst bucket item count to get to: 5000 curr count: 4378
2021/12/16 22:34:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 22:34:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 22:34:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:34:09 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 22:34:14 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 22:34:19 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 22:34:24 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 22:34:24 App: n1qlquery_consistency got un-deployed
2021/12/16 22:34:24 No eventing-con process running
2021/12/16 22:34:24 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/16 22:34:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:34:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 22:34:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:34:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 22:34:48 Waiting for dst bucket item count to get to: 0 curr count: 3513
2021/12/16 22:34:53 Waiting for dst bucket item count to get to: 0 curr count: 1013
2021/12/16 22:34:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.73s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1553.102s

N1ql tests took 0 hours, 25 minutes and 57 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/16 22:36:28 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:42766->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:42766->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/16 22:36:47 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 22:36:47 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/16 22:36:47 No eventing-con process running
2021/12/16 22:36:47 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/16 22:36:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:36:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:36:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:36:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:37:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:37:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:37:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:37:05 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:37:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 22:37:11 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 22:37:16 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 22:37:21 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 22:37:26 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 22:37:26 App: TestDuplicateWithEventingRebIn got deployed
2021/12/16 22:37:26 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 22:37:26 Starting bucket ops to source bucket
2021/12/16 22:37:41 Removing node(s): [] from the cluster
2021/12/16 22:37:46 Rebalance progress: 0
2021/12/16 22:37:51 Rebalance progress: 0
2021/12/16 22:37:56 Rebalance progress: 25
2021/12/16 22:38:01 Rebalance progress: 25
2021/12/16 22:38:06 Rebalance progress: 25
2021/12/16 22:38:11 Rebalance progress: 25
2021/12/16 22:38:16 Rebalance progress: 25
2021/12/16 22:38:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:38:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 22:38:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:38:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 22:38:26 Rebalance progress: 25
2021/12/16 22:38:31 Rebalance progress: 25
2021/12/16 22:38:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:38:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:38:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:38:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 22:38:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:38:36 Waiting for src bucket item count to get to: 200000 curr count: 99630
2021/12/16 22:38:41 Waiting for src bucket item count to get to: 200000 curr count: 99630
2021/12/16 22:38:46 Waiting for src bucket item count to get to: 200000 curr count: 116045
2021/12/16 22:38:51 Waiting for src bucket item count to get to: 200000 curr count: 116045
2021/12/16 22:38:56 Waiting for src bucket item count to get to: 200000 curr count: 132247
2021/12/16 22:39:01 Waiting for src bucket item count to get to: 200000 curr count: 132247
2021/12/16 22:39:06 Waiting for src bucket item count to get to: 200000 curr count: 148679
2021/12/16 22:39:11 Waiting for src bucket item count to get to: 200000 curr count: 148679
2021/12/16 22:39:16 Waiting for src bucket item count to get to: 200000 curr count: 165010
2021/12/16 22:39:21 Waiting for src bucket item count to get to: 200000 curr count: 165010
2021/12/16 22:39:26 Waiting for src bucket item count to get to: 200000 curr count: 180929
2021/12/16 22:39:31 Waiting for src bucket item count to get to: 200000 curr count: 180929
2021/12/16 22:39:36 Waiting for src bucket item count to get to: 200000 curr count: 197134
2021/12/16 22:39:41 Waiting for src bucket item count to get to: 200000 curr count: 197134
2021/12/16 22:39:51 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 22:39:56 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 22:40:01 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 22:40:06 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 22:40:11 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 22:40:16 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 22:40:21 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 22:40:26 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 22:40:31 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 22:40:36 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 22:40:36 Post SET ops. Expected item count: 200000 got 200001
2021/12/16 22:40:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:40:42 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 22:40:47 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 22:40:52 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 22:40:57 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 22:40:57 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/16 22:40:57 No eventing-con process running
2021/12/16 22:40:57 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/16 22:40:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:41:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:41:08 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 22:41:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:41:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:41:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:41:20 Waiting for dst bucket item count to get to: 0 curr count: 38318
2021/12/16 22:41:25 Waiting for dst bucket item count to get to: 0 curr count: 38318
2021/12/16 22:41:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (288.28s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/16 22:41:30 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 22:41:30 Removing node(s): [] from the cluster
2021/12/16 22:41:35 Rebalance progress: 100
2021/12/16 22:41:40 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 22:41:40 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/16 22:41:40 No eventing-con process running
2021/12/16 22:41:40 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/16 22:41:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:41:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:41:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:41:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:41:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:41:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:41:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:41:59 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:42:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 22:42:05 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 22:42:10 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 22:42:15 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 22:42:20 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 22:42:20 App: TestDuplicateWithEventingRebOut got deployed
2021/12/16 22:42:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 22:42:20 Starting bucket ops to source bucket
2021/12/16 22:42:25 Rebalance progress: 25
2021/12/16 22:42:30 Rebalance progress: 25
2021/12/16 22:42:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:42:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:42:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:42:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 22:42:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:42:35 Waiting for src bucket item count to get to: 200000 curr count: 14909
2021/12/16 22:42:40 Waiting for src bucket item count to get to: 200000 curr count: 31405
2021/12/16 22:42:45 Waiting for src bucket item count to get to: 200000 curr count: 31405
2021/12/16 22:42:50 Waiting for src bucket item count to get to: 200000 curr count: 47743
2021/12/16 22:42:55 Waiting for src bucket item count to get to: 200000 curr count: 47743
2021/12/16 22:43:00 Waiting for src bucket item count to get to: 200000 curr count: 64432
2021/12/16 22:43:05 Waiting for src bucket item count to get to: 200000 curr count: 64432
2021/12/16 22:43:10 Waiting for src bucket item count to get to: 200000 curr count: 64432
2021/12/16 22:43:15 Waiting for src bucket item count to get to: 200000 curr count: 80679
2021/12/16 22:43:20 Waiting for src bucket item count to get to: 200000 curr count: 80679
2021/12/16 22:43:25 Waiting for src bucket item count to get to: 200000 curr count: 96781
2021/12/16 22:43:30 Waiting for src bucket item count to get to: 200000 curr count: 96781
2021/12/16 22:43:35 Waiting for src bucket item count to get to: 200000 curr count: 113051
2021/12/16 22:43:40 Waiting for src bucket item count to get to: 200000 curr count: 113051
2021/12/16 22:43:45 Waiting for src bucket item count to get to: 200000 curr count: 129643
2021/12/16 22:43:50 Waiting for src bucket item count to get to: 200000 curr count: 129643
2021/12/16 22:43:55 Waiting for src bucket item count to get to: 200000 curr count: 146161
2021/12/16 22:44:00 Waiting for src bucket item count to get to: 200000 curr count: 146161
2021/12/16 22:44:05 Waiting for src bucket item count to get to: 200000 curr count: 162789
2021/12/16 22:44:10 Waiting for src bucket item count to get to: 200000 curr count: 162789
2021/12/16 22:44:15 Waiting for src bucket item count to get to: 200000 curr count: 179069
2021/12/16 22:44:20 Waiting for src bucket item count to get to: 200000 curr count: 179069
2021/12/16 22:44:25 Waiting for src bucket item count to get to: 200000 curr count: 195598
2021/12/16 22:44:30 Waiting for src bucket item count to get to: 200000 curr count: 195598
2021/12/16 22:44:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 22:44:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 22:44:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:44:40 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 22:44:45 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 22:44:50 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 22:44:55 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 22:44:55 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/16 22:44:55 No eventing-con process running
2021/12/16 22:44:55 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/16 22:44:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:45:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:45:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 22:45:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:45:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:45:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:45:19 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 22:45:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (233.77s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/16 22:45:29 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 22:45:29 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/16 22:45:29 No eventing-con process running
2021/12/16 22:45:29 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/16 22:45:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:45:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:45:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:45:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:45:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:45:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:45:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:45:47 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:45:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 22:45:54 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 22:45:59 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 22:46:04 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 22:46:09 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 22:46:09 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/16 22:46:09 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 22:46:09 Starting bucket ops to source bucket
2021/12/16 22:46:09 Removing node(s): [] from the cluster
2021/12/16 22:46:14 Rebalance progress: 25
2021/12/16 22:46:19 Rebalance progress: 25
2021/12/16 22:46:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:46:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:46:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:46:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 22:46:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:46:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 22:46:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 22:46:36 Rebalance progress: 16.66666666666667
2021/12/16 22:46:41 Rebalance progress: 16.66666666666667
2021/12/16 22:46:46 Rebalance progress: 16.66666666666667
2021/12/16 22:46:51 Rebalance progress: 81.8702290076336
2021/12/16 22:46:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:46:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:46:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:46:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:46:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 22:47:01 Rebalance progress: 16.66666666666667
2021/12/16 22:47:06 Rebalance progress: 16.66666666666667
2021/12/16 22:47:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:47:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:47:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:47:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:47:11 Waiting for src bucket item count to get to: 200000 curr count: 76123
2021/12/16 22:47:16 Waiting for src bucket item count to get to: 200000 curr count: 92440
2021/12/16 22:47:21 Waiting for src bucket item count to get to: 200000 curr count: 92440
2021/12/16 22:47:26 Waiting for src bucket item count to get to: 200000 curr count: 108187
2021/12/16 22:47:31 Waiting for src bucket item count to get to: 200000 curr count: 108187
2021/12/16 22:47:36 Waiting for src bucket item count to get to: 200000 curr count: 124759
2021/12/16 22:47:41 Waiting for src bucket item count to get to: 200000 curr count: 124759
2021/12/16 22:47:46 Waiting for src bucket item count to get to: 200000 curr count: 140793
2021/12/16 22:47:51 Waiting for src bucket item count to get to: 200000 curr count: 156838
2021/12/16 22:47:56 Waiting for src bucket item count to get to: 200000 curr count: 156838
2021/12/16 22:48:01 Waiting for src bucket item count to get to: 200000 curr count: 172839
2021/12/16 22:48:06 Waiting for src bucket item count to get to: 200000 curr count: 172839
2021/12/16 22:48:11 Waiting for src bucket item count to get to: 200000 curr count: 189124
2021/12/16 22:48:16 Waiting for src bucket item count to get to: 200000 curr count: 189124
2021/12/16 22:48:21 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 22:48:21 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 22:48:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:48:26 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 22:48:31 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 22:48:36 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 22:48:41 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 22:48:41 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/16 22:48:41 No eventing-con process running
2021/12/16 22:48:41 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/16 22:48:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:48:53 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 22:48:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 22:48:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:49:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:49:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:49:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (226.00s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/16 22:49:15 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 22:49:15 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/16 22:49:15 No eventing-con process running
2021/12/16 22:49:15 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/16 22:49:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:49:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:49:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:49:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:49:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:49:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:49:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:49:33 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:49:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 22:49:39 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 22:49:45 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 22:49:50 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 22:49:55 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 22:49:55 App: TestDuplicateWithKVRebIn got deployed
2021/12/16 22:49:55 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 22:49:55 Starting bucket ops to source bucket
2021/12/16 22:49:55 Removing node(s): [] from the cluster
2021/12/16 22:50:00 Rebalance progress: 16.66666666666667
2021/12/16 22:50:05 Rebalance progress: 16.66666666666667
2021/12/16 22:50:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:50:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:50:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:50:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:50:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 22:50:15 Rebalance progress: 16.66666666666667
2021/12/16 22:50:20 Rebalance progress: 16.66666666666667
2021/12/16 22:50:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:50:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:50:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:50:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:50:25 Waiting for src bucket item count to get to: 200000 curr count: 38079
2021/12/16 22:50:30 Waiting for src bucket item count to get to: 200000 curr count: 38079
2021/12/16 22:50:35 Waiting for src bucket item count to get to: 200000 curr count: 54000
2021/12/16 22:50:40 Waiting for src bucket item count to get to: 200000 curr count: 70001
2021/12/16 22:50:45 Waiting for src bucket item count to get to: 200000 curr count: 70001
2021/12/16 22:50:50 Waiting for src bucket item count to get to: 200000 curr count: 85255
2021/12/16 22:50:55 Waiting for src bucket item count to get to: 200000 curr count: 85255
2021/12/16 22:51:00 Waiting for src bucket item count to get to: 200000 curr count: 101589
2021/12/16 22:51:05 Waiting for src bucket item count to get to: 200000 curr count: 101589
2021/12/16 22:51:10 Waiting for src bucket item count to get to: 200000 curr count: 118063
2021/12/16 22:51:15 Waiting for src bucket item count to get to: 200000 curr count: 118063
2021/12/16 22:51:20 Waiting for src bucket item count to get to: 200000 curr count: 134403
2021/12/16 22:51:25 Waiting for src bucket item count to get to: 200000 curr count: 134403
2021/12/16 22:51:30 Waiting for src bucket item count to get to: 200000 curr count: 134403
2021/12/16 22:51:35 Waiting for src bucket item count to get to: 200000 curr count: 150791
2021/12/16 22:51:40 Waiting for src bucket item count to get to: 200000 curr count: 150791
2021/12/16 22:51:45 Waiting for src bucket item count to get to: 200000 curr count: 166775
2021/12/16 22:51:50 Waiting for src bucket item count to get to: 200000 curr count: 166775
2021/12/16 22:51:55 Waiting for src bucket item count to get to: 200000 curr count: 181736
2021/12/16 22:52:00 Waiting for src bucket item count to get to: 200000 curr count: 181736
2021/12/16 22:52:05 Waiting for src bucket item count to get to: 200000 curr count: 198061
2021/12/16 22:52:10 Waiting for src bucket item count to get to: 200000 curr count: 198061
2021/12/16 22:52:15 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 22:52:15 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 22:52:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:52:20 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 22:52:26 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 22:52:31 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 22:52:36 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 22:52:36 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/16 22:52:36 No eventing-con process running
2021/12/16 22:52:36 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/16 22:52:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:52:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:52:47 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 22:52:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:52:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:52:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:52:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 22:52:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (234.27s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/16 22:53:04 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 22:53:05 Removing node(s): [] from the cluster
2021/12/16 22:53:10 Rebalance progress: 100
2021/12/16 22:53:15 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 22:53:15 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/16 22:53:15 No eventing-con process running
2021/12/16 22:53:15 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/16 22:53:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:53:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:53:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:53:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:53:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:53:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:53:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:53:33 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:53:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 22:53:39 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/16 22:53:44 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/16 22:53:49 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/16 22:53:54 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/16 22:53:54 App: TestDuplicateWithKVRebOut got deployed
2021/12/16 22:53:54 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 22:53:54 Starting bucket ops to source bucket
2021/12/16 22:53:59 Rebalance progress: 16.66666666666667
2021/12/16 22:54:04 Rebalance progress: 16.66666666666667
2021/12/16 22:54:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:54:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:54:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:54:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:54:09 Waiting for src bucket item count to get to: 200000 curr count: 7309
2021/12/16 22:54:14 Waiting for src bucket item count to get to: 200000 curr count: 23188
2021/12/16 22:54:19 Waiting for src bucket item count to get to: 200000 curr count: 23188
2021/12/16 22:54:24 Waiting for src bucket item count to get to: 200000 curr count: 39234
2021/12/16 22:54:29 Waiting for src bucket item count to get to: 200000 curr count: 39234
2021/12/16 22:54:34 Waiting for src bucket item count to get to: 200000 curr count: 55336
2021/12/16 22:54:39 Waiting for src bucket item count to get to: 200000 curr count: 55336
2021/12/16 22:54:44 Waiting for src bucket item count to get to: 200000 curr count: 55336
2021/12/16 22:54:49 Waiting for src bucket item count to get to: 200000 curr count: 71360
2021/12/16 22:54:54 Waiting for src bucket item count to get to: 200000 curr count: 71360
2021/12/16 22:54:59 Waiting for src bucket item count to get to: 200000 curr count: 86417
2021/12/16 22:55:04 Waiting for src bucket item count to get to: 200000 curr count: 102335
2021/12/16 22:55:09 Waiting for src bucket item count to get to: 200000 curr count: 102335
2021/12/16 22:55:14 Waiting for src bucket item count to get to: 200000 curr count: 118339
2021/12/16 22:55:19 Waiting for src bucket item count to get to: 200000 curr count: 118339
2021/12/16 22:55:24 Waiting for src bucket item count to get to: 200000 curr count: 134253
2021/12/16 22:55:29 Waiting for src bucket item count to get to: 200000 curr count: 134253
2021/12/16 22:55:34 Waiting for src bucket item count to get to: 200000 curr count: 150246
2021/12/16 22:55:39 Waiting for src bucket item count to get to: 200000 curr count: 150246
2021/12/16 22:55:44 Waiting for src bucket item count to get to: 200000 curr count: 166327
2021/12/16 22:55:49 Waiting for src bucket item count to get to: 200000 curr count: 166327
2021/12/16 22:55:54 Waiting for src bucket item count to get to: 200000 curr count: 181325
2021/12/16 22:55:59 Waiting for src bucket item count to get to: 200000 curr count: 181325
2021/12/16 22:56:04 Waiting for src bucket item count to get to: 200000 curr count: 197177
2021/12/16 22:56:09 Waiting for src bucket item count to get to: 200000 curr count: 197177
2021/12/16 22:56:14 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 22:56:14 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 22:56:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 22:56:20 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 22:56:25 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 22:56:30 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 22:56:35 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 22:56:35 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/16 22:56:35 No eventing-con process running
2021/12/16 22:56:35 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/16 22:56:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:56:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:56:46 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 22:56:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:56:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:56:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:56:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 22:56:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (239.05s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/16 22:57:08 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 22:57:08 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/16 22:57:08 No eventing-con process running
2021/12/16 22:57:08 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/16 22:57:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:57:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:57:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 22:57:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:57:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:57:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:57:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 22:57:27 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 22:57:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 22:57:33 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 22:57:38 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 22:57:43 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 22:57:48 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 22:57:48 App: TestDuplicateWithKVSwapReb got deployed
2021/12/16 22:57:48 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 22:57:48 Starting bucket ops to source bucket
2021/12/16 22:57:48 Removing node(s): [] from the cluster
2021/12/16 22:57:53 Rebalance progress: 16.66666666666667
2021/12/16 22:57:58 Rebalance progress: 16.66666666666667
2021/12/16 22:58:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:58:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:58:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:58:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:58:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 22:58:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 22:58:09 Rebalance progress: 16.66666666666667
2021/12/16 22:58:14 Rebalance progress: 16.66666666666667
2021/12/16 22:58:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:58:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:58:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:58:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:58:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 22:58:24 Rebalance progress: 16.66666666666667
2021/12/16 22:58:29 Rebalance progress: 16.66666666666667
2021/12/16 22:58:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:58:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 22:58:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 22:58:34 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 22:58:34 Waiting for src bucket item count to get to: 200000 curr count: 64906
2021/12/16 22:58:39 Waiting for src bucket item count to get to: 200000 curr count: 64906
2021/12/16 22:58:44 Waiting for src bucket item count to get to: 200000 curr count: 80159
2021/12/16 22:58:49 Waiting for src bucket item count to get to: 200000 curr count: 80159
2021/12/16 22:58:54 Waiting for src bucket item count to get to: 200000 curr count: 96352
2021/12/16 22:58:59 Waiting for src bucket item count to get to: 200000 curr count: 96352
2021/12/16 22:59:04 Waiting for src bucket item count to get to: 200000 curr count: 112734
2021/12/16 22:59:09 Waiting for src bucket item count to get to: 200000 curr count: 112734
2021/12/16 22:59:14 Waiting for src bucket item count to get to: 200000 curr count: 128889
2021/12/16 22:59:19 Waiting for src bucket item count to get to: 200000 curr count: 128889
2021/12/16 22:59:24 Waiting for src bucket item count to get to: 200000 curr count: 128889
2021/12/16 22:59:29 Waiting for src bucket item count to get to: 200000 curr count: 145390
2021/12/16 22:59:34 Waiting for src bucket item count to get to: 200000 curr count: 145390
2021/12/16 22:59:39 Waiting for src bucket item count to get to: 200000 curr count: 161827
2021/12/16 22:59:44 Waiting for src bucket item count to get to: 200000 curr count: 161827
2021/12/16 22:59:49 Waiting for src bucket item count to get to: 200000 curr count: 177252
2021/12/16 22:59:54 Waiting for src bucket item count to get to: 200000 curr count: 177252
2021/12/16 22:59:59 Waiting for src bucket item count to get to: 200000 curr count: 193387
2021/12/16 23:00:04 Waiting for src bucket item count to get to: 200000 curr count: 193387
2021/12/16 23:00:09 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 23:00:09 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 23:00:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:00:14 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 23:00:19 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 23:00:24 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 23:00:29 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 23:00:29 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/16 23:00:29 No eventing-con process running
2021/12/16 23:00:29 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/16 23:00:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:00:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:00:41 Waiting for dst bucket item count to get to: 0 curr count: 165832
2021/12/16 23:00:46 Waiting for dst bucket item count to get to: 0 curr count: 165832
2021/12/16 23:00:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:00:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:00:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:00:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 23:00:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (239.44s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/16 23:01:08 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:01:08 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 23:01:08 No eventing-con process running
2021/12/16 23:01:08 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/16 23:01:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:01:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:01:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:01:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:01:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:01:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:01:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:01:26 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:01:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/16 23:01:32 Starting bucket ops to source bucket
2021/12/16 23:01:37 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 23:01:42 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 23:01:47 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 23:01:47 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:01:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:01:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:01:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 23:01:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:01:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:01:53 Rebalance progress: 16.66666666666667
2021/12/16 23:01:58 Rebalance progress: 16.66666666666667
2021/12/16 23:02:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:02:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:02:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:02:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 23:02:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:02:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:02:08 Rebalance progress: 16.66666666666667
2021/12/16 23:02:13 Rebalance progress: 16.66666666666667
2021/12/16 23:02:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:02:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:02:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:02:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 23:02:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 23:02:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:02:29 Rebalance progress: 12.5
2021/12/16 23:02:34 Rebalance progress: 12.5
2021/12/16 23:02:39 Rebalance progress: 12.5
2021/12/16 23:02:44 Rebalance progress: 98.55371900826447
2021/12/16 23:02:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:02:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:02:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:02:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:02:54 Rebalance progress: 12.5
2021/12/16 23:02:59 Rebalance progress: 12.5
2021/12/16 23:03:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:03:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:03:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:03:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 23:03:09 Rebalance progress: 12.5
2021/12/16 23:03:14 Rebalance progress: 12.5
2021/12/16 23:03:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:03:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:03:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:03:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 23:03:24 Rebalance progress: 12.5
2021/12/16 23:03:29 Rebalance progress: 12.5
2021/12/16 23:03:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:03:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:03:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:03:34 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:03:39 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:03:44 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:03:49 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:03:54 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:03:54 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 23:03:54 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:03:54 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/16 23:03:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:04:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:04:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12153
2021/12/16 23:04:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:04:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:04:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:04:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12153
2021/12/16 23:04:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.13s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/16 23:04:28 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 23:04:28 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/16 23:04:28 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:04:28 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/16 23:04:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:04:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:04:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:04:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:04:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:04:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:04:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:04:46 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:04:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/16 23:04:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 23:04:57 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/16 23:05:02 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/16 23:05:07 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/16 23:05:07 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:05:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:05:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:05:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:05:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:05:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 23:05:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:05:14 Rebalance progress: 12.5
2021/12/16 23:05:19 Rebalance progress: 12.5
2021/12/16 23:05:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:05:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:05:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:05:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 23:05:29 Rebalance progress: 12.5
2021/12/16 23:05:34 Rebalance progress: 12.5
2021/12/16 23:05:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:05:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:05:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:05:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:05:44 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 23:05:49 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 23:05:54 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 23:05:59 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 23:05:59 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/16 23:05:59 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:05:59 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/16 23:06:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:06:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:06:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4653
2021/12/16 23:06:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4653
2021/12/16 23:06:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:06:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:06:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:06:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4653
2021/12/16 23:06:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (130.12s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/16 23:06:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 23:06:38 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 23:06:38 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:06:38 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/16 23:06:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:06:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:06:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:06:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:06:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:06:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:06:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:06:57 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:06:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 23:07:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 23:07:08 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/16 23:07:13 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/16 23:07:18 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/16 23:07:18 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:07:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:07:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:07:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:07:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:07:23 Rebalance progress: 12.5
2021/12/16 23:07:28 Rebalance progress: 12.5
2021/12/16 23:07:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:07:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:07:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:07:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:07:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:07:39 Rebalance progress: 12.5
2021/12/16 23:07:44 Rebalance progress: 12.5
2021/12/16 23:07:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:07:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:07:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:07:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 23:07:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:07:55 Rebalance progress: 12.5
2021/12/16 23:08:00 Rebalance progress: 12.5
2021/12/16 23:08:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:08:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:08:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:08:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:08:10 Rebalance progress: 12.5
2021/12/16 23:08:15 Rebalance progress: 12.5
2021/12/16 23:08:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:08:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:08:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:08:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 23:08:25 Rebalance progress: 12.5
2021/12/16 23:08:30 Rebalance progress: 12.5
2021/12/16 23:08:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:08:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 23:08:40 Rebalance progress: 12.5
2021/12/16 23:08:45 Rebalance progress: 12.5
2021/12/16 23:08:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:08:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:08:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:08:50 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:08:55 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 23:09:00 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 23:09:05 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 23:09:10 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 23:09:10 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 23:09:10 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:09:10 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/16 23:09:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:09:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:09:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10709
2021/12/16 23:09:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10709
2021/12/16 23:09:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:09:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:09:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:09:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10708
2021/12/16 23:09:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.73s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/16 23:09:49 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 23:09:49 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/16 23:09:49 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:09:49 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/16 23:09:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:09:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:09:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:09:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:10:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:10:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:10:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:10:02 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:10:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 23:10:08 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 23:10:13 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/16 23:10:18 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/16 23:10:23 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/16 23:10:23 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:10:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:10:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:10:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:10:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:10:29 Rebalance progress: 12.5
2021/12/16 23:10:34 Rebalance progress: 12.5
2021/12/16 23:10:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:10:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:10:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:10:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:10:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:10:44 Rebalance progress: 12.5
2021/12/16 23:10:49 Rebalance progress: 12.5
2021/12/16 23:10:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:10:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:10:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:10:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 23:11:00 Rebalance progress: 12.5
2021/12/16 23:11:05 Rebalance progress: 12.5
2021/12/16 23:11:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:11:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:11:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:11:10 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:11:15 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 23:11:20 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 23:11:25 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 23:11:30 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 23:11:30 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/16 23:11:30 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:11:30 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/16 23:11:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:11:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:11:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6118
2021/12/16 23:11:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6118
2021/12/16 23:11:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:11:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:11:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:11:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6118
2021/12/16 23:11:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.74s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/16 23:12:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:12:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:12:09 Rebalance progress: 100
2021/12/16 23:12:14 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 23:12:14 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/16 23:12:14 No eventing-con process running
2021/12/16 23:12:14 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/16 23:12:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:12:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:12:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:12:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:12:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:12:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:12:27 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:12:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/16 23:12:33 Starting bucket ops to source bucket
2021/12/16 23:12:38 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/16 23:12:43 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/16 23:12:48 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/16 23:12:48 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:12:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:12:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:12:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:12:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:12:54 Rebalance progress: 12.5
2021/12/16 23:12:59 Rebalance progress: 12.5
2021/12/16 23:13:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:13:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:13:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:13:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/16 23:13:04 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/16 23:13:09 Rebalance progress: 12.5
2021/12/16 23:13:14 Rebalance progress: 12.5
2021/12/16 23:13:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:13:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:13:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:13:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:13:24 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 23:13:29 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 23:13:34 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 23:13:39 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 23:13:39 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/16 23:13:39 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:13:39 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/16 23:13:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:13:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:13:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3700
2021/12/16 23:13:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3700
2021/12/16 23:13:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:13:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:14:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:14:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4526
2021/12/16 23:14:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.19s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/16 23:14:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:14:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:14:18 Rebalance progress: 100
2021/12/16 23:14:23 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 23:14:23 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/16 23:14:23 No eventing-con process running
2021/12/16 23:14:23 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/16 23:14:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:14:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:14:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:14:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:14:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:14:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:14:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:14:37 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:14:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/16 23:14:43 Starting bucket ops to source bucket
2021/12/16 23:14:48 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/16 23:14:53 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/16 23:14:58 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/16 23:14:58 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:14:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:14:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:14:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:14:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:15:03 Rebalance progress: 12.5
2021/12/16 23:15:08 Rebalance progress: 12.5
2021/12/16 23:15:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:15:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:15:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:15:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/16 23:15:13 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/16 23:15:18 Rebalance progress: 12.5
2021/12/16 23:15:23 Rebalance progress: 12.5
2021/12/16 23:15:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:15:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:15:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:15:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:15:29 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/16 23:15:34 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 23:15:39 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 23:15:44 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 23:15:49 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 23:15:49 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/16 23:15:49 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:15:49 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/16 23:15:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:15:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:16:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 18817
2021/12/16 23:16:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 18817
2021/12/16 23:16:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:16:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:16:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:16:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37771
2021/12/16 23:16:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.70s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/16 23:16:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 23:16:27 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/16 23:16:27 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:16:27 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/16 23:16:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:16:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:16:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:16:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:16:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:16:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:16:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:16:46 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:16:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:16:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:16:47 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:16:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 23:16:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 23:16:58 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/16 23:17:03 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/16 23:17:08 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/16 23:17:08 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/16 23:17:13 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/16 23:17:18 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/16 23:17:23 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/16 23:17:23 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:17:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:17:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:17:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:17:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:17:28 Rebalance progress: 12.5
2021/12/16 23:17:33 Rebalance progress: 12.5
2021/12/16 23:17:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:17:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:17:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:17:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:17:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:17:44 Rebalance progress: 12.5
2021/12/16 23:17:49 Rebalance progress: 12.5
2021/12/16 23:17:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:17:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:17:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:17:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 23:17:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:17:59 Rebalance progress: 12.5
2021/12/16 23:18:04 Rebalance progress: 12.5
2021/12/16 23:18:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:18:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:18:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:18:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:18:14 Rebalance progress: 12.5
2021/12/16 23:18:19 Rebalance progress: 12.5
2021/12/16 23:18:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:18:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:18:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:18:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 23:18:29 Rebalance progress: 12.5
2021/12/16 23:18:34 Rebalance progress: 12.5
2021/12/16 23:18:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:18:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:18:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:18:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 23:18:44 Rebalance progress: 12.5
2021/12/16 23:18:49 Rebalance progress: 12.5
2021/12/16 23:18:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:18:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:18:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:18:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:19:00 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 23:19:05 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 23:19:10 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 23:19:15 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 23:19:15 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/16 23:19:15 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:19:15 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/16 23:19:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:19:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:19:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12126
2021/12/16 23:19:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:19:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:19:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:19:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:19:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:19:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 23:19:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 23:19:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 23:19:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 23:19:55 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/16 23:19:55 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:19:55 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/16 23:19:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:20:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:20:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:20:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:20:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:20:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (231.23s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/16 23:20:18 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 23:20:19 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/16 23:20:19 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:20:19 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/16 23:20:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:20:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:20:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:20:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:20:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:20:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:20:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:20:37 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:20:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:20:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:20:38 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:20:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 23:20:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 23:20:49 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/16 23:20:54 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/16 23:20:59 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/16 23:20:59 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/16 23:21:04 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/16 23:21:09 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/16 23:21:14 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/16 23:21:14 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:21:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:21:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:21:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:21:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:21:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 23:21:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:21:21 Rebalance progress: 12.5
2021/12/16 23:21:26 Rebalance progress: 12.5
2021/12/16 23:21:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:21:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:21:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:21:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 23:21:36 Rebalance progress: 12.5
2021/12/16 23:21:41 Rebalance progress: 12.5
2021/12/16 23:21:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:21:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:21:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:21:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:21:51 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 23:21:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 23:22:01 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 23:22:06 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 23:22:06 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/16 23:22:06 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:22:06 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/16 23:22:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:22:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:22:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:22:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:22:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:22:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6171
2021/12/16 23:22:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:22:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:22:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 23:22:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 23:22:42 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 23:22:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 23:22:47 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/16 23:22:47 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:22:47 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/16 23:22:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:22:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:22:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:22:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:23:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:23:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (171.67s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/16 23:23:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 23:23:10 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/16 23:23:10 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:23:10 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/16 23:23:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:23:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:23:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:23:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:23:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:23:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:23:29 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 23:23:29 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/16 23:23:29 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:23:29 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/16 23:23:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:23:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:23:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:23:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:23:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:23:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:23:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:23:42 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:23:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:23:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:23:44 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:23:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/16 23:23:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 23:23:55 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/16 23:24:00 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/16 23:24:05 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/16 23:24:05 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/16 23:24:10 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/16 23:24:15 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/16 23:24:20 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/16 23:24:20 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:24:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:24:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:24:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:24:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:24:25 Rebalance progress: 12.5
2021/12/16 23:24:30 Rebalance progress: 12.5
2021/12/16 23:24:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:24:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:24:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:24:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:24:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:24:41 Rebalance progress: 12.5
2021/12/16 23:24:46 Rebalance progress: 12.5
2021/12/16 23:24:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:24:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:24:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:24:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 23:24:56 Rebalance progress: 12.5
2021/12/16 23:25:01 Rebalance progress: 12.5
2021/12/16 23:25:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:25:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:25:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:25:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:25:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 23:25:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 23:25:21 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 23:25:26 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 23:25:26 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/16 23:25:26 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:25:26 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/16 23:25:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:25:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:25:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:25:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:25:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:25:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7602
2021/12/16 23:25:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:25:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:25:52 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 23:25:57 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 23:26:02 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 23:26:07 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 23:26:07 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/16 23:26:07 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:26:07 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/16 23:26:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:26:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:26:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:26:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:26:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:26:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (200.31s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/16 23:26:30 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:26:30 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 23:26:30 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:26:30 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/16 23:26:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:26:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:26:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:26:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:26:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:26:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:26:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:26:49 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:26:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/16 23:26:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 23:27:00 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 23:27:05 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 23:27:10 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 23:27:10 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:27:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:27:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:27:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:27:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:27:31 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:27:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:27:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:28:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:28:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:28:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:28:06 Rebalance progress: 12.5
2021/12/16 23:28:11 Rebalance progress: 12.5
2021/12/16 23:28:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:28:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:28:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:28:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:28:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:28:22 Rebalance progress: 12.5
2021/12/16 23:28:27 Rebalance progress: 12.5
2021/12/16 23:28:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:28:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:28:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:28:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 23:28:37 Rebalance progress: 12.5
2021/12/16 23:28:42 Rebalance progress: 12.5
2021/12/16 23:28:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:28:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:28:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:28:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:28:52 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:28:57 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:29:02 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:29:07 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:29:07 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 23:29:07 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:29:07 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/16 23:29:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:29:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:29:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11154
2021/12/16 23:29:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:29:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:29:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:29:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7217
2021/12/16 23:29:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7217
2021/12/16 23:29:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.13s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/16 23:29:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:29:46 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 23:29:46 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:29:46 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/16 23:29:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:29:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:29:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:29:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:29:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:29:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:30:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:30:04 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:30:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/16 23:30:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 23:30:15 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 23:30:20 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 23:30:25 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 23:30:25 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:30:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:30:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:30:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:30:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:30:46 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:30:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:30:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:30:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:30:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:30:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:30:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:31:17 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:31:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:31:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:31:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:31:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:31:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:31:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:31:48 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:31:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:31:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:31:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:31:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:31:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:31:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:32:19 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:32:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:32:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:32:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:32:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:32:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:32:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:32:50 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:32:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:32:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:33:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:33:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:33:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:33:05 Rebalance progress: 12.5
2021/12/16 23:33:10 Rebalance progress: 12.5
2021/12/16 23:33:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:33:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:33:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:33:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:33:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:33:21 Rebalance progress: 12.5
2021/12/16 23:33:26 Rebalance progress: 12.5
2021/12/16 23:33:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:33:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:33:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:33:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 23:33:36 Rebalance progress: 12.5
2021/12/16 23:33:41 Rebalance progress: 12.5
2021/12/16 23:33:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:33:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:33:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:33:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:33:51 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:33:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:34:01 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:34:06 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 23:34:06 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 23:34:06 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:34:06 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/16 23:34:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:34:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:34:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21555
2021/12/16 23:34:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:34:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:34:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:34:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (289.18s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/16 23:34:35 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 23:34:35 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/16 23:34:35 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:34:35 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/16 23:34:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:34:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:34:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:34:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:34:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:34:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:34:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:34:53 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:34:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 23:34:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 23:35:04 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/16 23:35:09 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/16 23:35:14 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/16 23:35:14 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:35:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:35:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:35:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:35:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:35:20 Rebalance progress: 12.5
2021/12/16 23:35:25 Rebalance progress: 12.5
2021/12/16 23:35:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:35:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:35:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:35:30 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:35:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:35:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:35:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:35:45 Rebalance progress: 12.5
2021/12/16 23:35:50 Rebalance progress: 12.5
2021/12/16 23:35:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:35:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:35:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:35:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:35:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:36:01 Rebalance progress: 12.5
2021/12/16 23:36:06 Rebalance progress: 12.5
2021/12/16 23:36:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:36:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:36:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:36:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 23:36:16 Rebalance progress: 12.5
2021/12/16 23:36:21 Rebalance progress: 12.5
2021/12/16 23:36:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:36:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:36:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:36:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:36:31 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 23:36:36 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 23:36:41 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 23:36:46 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 23:36:46 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/16 23:36:46 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:36:46 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/16 23:36:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:36:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:36:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8656
2021/12/16 23:36:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:37:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:37:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:37:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (160.06s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/16 23:37:15 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 23:37:15 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/16 23:37:15 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:37:15 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/16 23:37:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:37:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:37:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:37:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:37:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:37:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:37:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:37:33 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:37:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 23:37:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 23:37:44 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/16 23:37:49 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/16 23:37:54 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/16 23:37:55 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:37:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:37:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:37:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/16 23:37:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:38:00 Rebalance progress: 12.5
2021/12/16 23:38:05 Rebalance progress: 12.5
2021/12/16 23:38:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:38:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:38:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:38:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/16 23:38:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:38:16 Rebalance progress: 12.5
2021/12/16 23:38:21 Rebalance progress: 12.5
2021/12/16 23:38:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:38:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:38:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:38:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 23:38:31 Rebalance progress: 12.5
2021/12/16 23:38:36 Rebalance progress: 12.5
2021/12/16 23:38:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:38:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:38:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:38:41 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:38:46 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 23:38:51 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 23:38:56 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 23:39:01 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 23:39:01 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/16 23:39:01 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:39:01 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/16 23:39:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:39:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:39:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6108
2021/12/16 23:39:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6108
2021/12/16 23:39:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:39:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:39:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:39:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.63s)
=== RUN   TestEventingRebInWithOps
2021/12/16 23:39:34 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 23:39:34 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/16 23:39:34 eventingSetup.go:46: No eventing-con process running
2021/12/16 23:39:34 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/16 23:39:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:39:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:39:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:39:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:39:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:39:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:39:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:39:53 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:39:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 23:39:59 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 23:40:04 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 23:40:09 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 23:40:14 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 23:40:14 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/16 23:40:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:40:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 23:40:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 23:40:20 Rebalance progress: 12.5
2021/12/16 23:40:25 Rebalance progress: 12.5
2021/12/16 23:40:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:40:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:40:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:40:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:40:35 Rebalance progress: 12.5
2021/12/16 23:40:40 Rebalance progress: 12.5
2021/12/16 23:40:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:40:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:40:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:40:45 Waiting for src bucket item count to get to: 200000 curr count: 38358
2021/12/16 23:40:50 Waiting for src bucket item count to get to: 200000 curr count: 38358
2021/12/16 23:40:55 Waiting for src bucket item count to get to: 200000 curr count: 54318
2021/12/16 23:41:00 Waiting for src bucket item count to get to: 200000 curr count: 54318
2021/12/16 23:41:05 Waiting for src bucket item count to get to: 200000 curr count: 70141
2021/12/16 23:41:10 Waiting for src bucket item count to get to: 200000 curr count: 70141
2021/12/16 23:41:15 Waiting for src bucket item count to get to: 200000 curr count: 84746
2021/12/16 23:41:20 Waiting for src bucket item count to get to: 200000 curr count: 101030
2021/12/16 23:41:25 Waiting for src bucket item count to get to: 200000 curr count: 101030
2021/12/16 23:41:30 Waiting for src bucket item count to get to: 200000 curr count: 101030
2021/12/16 23:41:35 Waiting for src bucket item count to get to: 200000 curr count: 117293
2021/12/16 23:41:40 Waiting for src bucket item count to get to: 200000 curr count: 117293
2021/12/16 23:41:45 Waiting for src bucket item count to get to: 200000 curr count: 133418
2021/12/16 23:41:50 Waiting for src bucket item count to get to: 200000 curr count: 133418
2021/12/16 23:41:55 Waiting for src bucket item count to get to: 200000 curr count: 149685
2021/12/16 23:42:00 Waiting for src bucket item count to get to: 200000 curr count: 149685
2021/12/16 23:42:05 Waiting for src bucket item count to get to: 200000 curr count: 165463
2021/12/16 23:42:10 Waiting for src bucket item count to get to: 200000 curr count: 165463
2021/12/16 23:42:15 Waiting for src bucket item count to get to: 200000 curr count: 180413
2021/12/16 23:42:20 Waiting for src bucket item count to get to: 200000 curr count: 180413
2021/12/16 23:42:25 Waiting for src bucket item count to get to: 200000 curr count: 196507
2021/12/16 23:42:30 Waiting for src bucket item count to get to: 200000 curr count: 196507
2021/12/16 23:42:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 23:42:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 23:42:35 Starting bucket ops to source bucket
2021/12/16 23:46:25 Waiting for dst bucket item count to get to: 0 curr count: 19528
2021/12/16 23:46:30 Waiting for dst bucket item count to get to: 0 curr count: 1125
2021/12/16 23:46:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:46:30 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 23:46:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:46:35 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 23:46:40 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 23:46:45 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 23:46:50 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 23:46:50 App: TestEventingRebInWithOps got un-deployed
2021/12/16 23:46:50 No eventing-con process running
2021/12/16 23:46:50 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/16 23:46:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:46:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:46:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:46:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:47:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:47:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (459.19s)
=== RUN   TestEventingRebOutWithOps
2021/12/16 23:47:09 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:47:09 Removing node(s): [] from the cluster
2021/12/16 23:47:14 Rebalance progress: 100
2021/12/16 23:47:19 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 23:47:19 App: TestEventingRebOutWithOps got un-deployed
2021/12/16 23:47:19 No eventing-con process running
2021/12/16 23:47:19 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/16 23:47:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:47:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:47:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:47:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:47:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:47:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:47:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:47:37 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:47:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 23:47:44 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 23:47:49 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 23:47:54 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 23:47:59 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 23:47:59 App: TestEventingRebOutWithOps got deployed
2021/12/16 23:47:59 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:47:59 Starting bucket ops to source bucket
2021/12/16 23:48:04 Rebalance progress: 12.5
2021/12/16 23:48:09 Rebalance progress: 12.5
2021/12/16 23:48:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:48:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:48:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:48:14 Waiting for src bucket item count to get to: 200000 curr count: 16269
2021/12/16 23:48:19 Waiting for src bucket item count to get to: 200000 curr count: 16269
2021/12/16 23:48:24 Waiting for src bucket item count to get to: 200000 curr count: 32167
2021/12/16 23:48:29 Waiting for src bucket item count to get to: 200000 curr count: 32167
2021/12/16 23:48:34 Waiting for src bucket item count to get to: 200000 curr count: 48338
2021/12/16 23:48:39 Waiting for src bucket item count to get to: 200000 curr count: 48338
2021/12/16 23:48:44 Waiting for src bucket item count to get to: 200000 curr count: 48338
2021/12/16 23:48:49 Waiting for src bucket item count to get to: 200000 curr count: 64603
2021/12/16 23:48:54 Waiting for src bucket item count to get to: 200000 curr count: 64603
2021/12/16 23:48:59 Waiting for src bucket item count to get to: 200000 curr count: 79858
2021/12/16 23:49:04 Waiting for src bucket item count to get to: 200000 curr count: 79858
2021/12/16 23:49:09 Waiting for src bucket item count to get to: 200000 curr count: 95855
2021/12/16 23:49:14 Waiting for src bucket item count to get to: 200000 curr count: 95855
2021/12/16 23:49:19 Waiting for src bucket item count to get to: 200000 curr count: 112133
2021/12/16 23:49:24 Waiting for src bucket item count to get to: 200000 curr count: 112133
2021/12/16 23:49:29 Waiting for src bucket item count to get to: 200000 curr count: 128017
2021/12/16 23:49:34 Waiting for src bucket item count to get to: 200000 curr count: 143931
2021/12/16 23:49:39 Waiting for src bucket item count to get to: 200000 curr count: 143931
2021/12/16 23:49:44 Waiting for src bucket item count to get to: 200000 curr count: 159750
2021/12/16 23:49:49 Waiting for src bucket item count to get to: 200000 curr count: 159750
2021/12/16 23:49:54 Waiting for src bucket item count to get to: 200000 curr count: 174570
2021/12/16 23:49:59 Waiting for src bucket item count to get to: 200000 curr count: 174570
2021/12/16 23:50:04 Waiting for src bucket item count to get to: 200000 curr count: 190211
2021/12/16 23:50:09 Waiting for src bucket item count to get to: 200000 curr count: 190211
2021/12/16 23:50:14 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 23:50:14 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 23:50:14 Starting bucket ops to source bucket
2021/12/16 23:54:04 Waiting for dst bucket item count to get to: 0 curr count: 17340
2021/12/16 23:54:09 Waiting for dst bucket item count to get to: 0 curr count: 17340
2021/12/16 23:54:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:54:09 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 23:54:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 23:54:14 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 23:54:19 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 23:54:24 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 23:54:29 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 23:54:29 App: TestEventingRebOutWithOps got un-deployed
2021/12/16 23:54:29 No eventing-con process running
2021/12/16 23:54:29 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/16 23:54:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:54:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:54:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:54:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:54:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:54:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (458.98s)
=== RUN   TestEventingSwapRebWithOps
2021/12/16 23:54:53 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 23:54:53 App: TestEventingSwapRebWithOps got un-deployed
2021/12/16 23:54:53 No eventing-con process running
2021/12/16 23:54:53 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/16 23:54:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:54:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:54:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 23:55:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:55:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:55:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:55:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 23:55:11 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 23:55:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 23:55:17 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 23:55:22 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 23:55:27 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 23:55:32 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 23:55:32 App: TestEventingSwapRebWithOps got deployed
2021/12/16 23:55:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 23:55:32 Starting bucket ops to source bucket
2021/12/16 23:55:33 Removing node(s): [] from the cluster
2021/12/16 23:55:38 Rebalance progress: 12.5
2021/12/16 23:55:43 Rebalance progress: 12.5
2021/12/16 23:55:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:55:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:55:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:55:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 23:55:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 23:55:54 Rebalance progress: 12.5
2021/12/16 23:55:59 Rebalance progress: 12.5
2021/12/16 23:56:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:56:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:56:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:56:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 23:56:09 Rebalance progress: 12.5
2021/12/16 23:56:14 Rebalance progress: 12.5
2021/12/16 23:56:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:56:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 23:56:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 23:56:19 Waiting for src bucket item count to get to: 200000 curr count: 55732
2021/12/16 23:56:24 Waiting for src bucket item count to get to: 200000 curr count: 70651
2021/12/16 23:56:29 Waiting for src bucket item count to get to: 200000 curr count: 70651
2021/12/16 23:56:34 Waiting for src bucket item count to get to: 200000 curr count: 86245
2021/12/16 23:56:39 Waiting for src bucket item count to get to: 200000 curr count: 86245
2021/12/16 23:56:44 Waiting for src bucket item count to get to: 200000 curr count: 102477
2021/12/16 23:56:49 Waiting for src bucket item count to get to: 200000 curr count: 102477
2021/12/16 23:56:54 Waiting for src bucket item count to get to: 200000 curr count: 118435
2021/12/16 23:56:59 Waiting for src bucket item count to get to: 200000 curr count: 118435
2021/12/16 23:57:04 Waiting for src bucket item count to get to: 200000 curr count: 134199
2021/12/16 23:57:09 Waiting for src bucket item count to get to: 200000 curr count: 134199
2021/12/16 23:57:14 Waiting for src bucket item count to get to: 200000 curr count: 150005
2021/12/16 23:57:19 Waiting for src bucket item count to get to: 200000 curr count: 150005
2021/12/16 23:57:24 Waiting for src bucket item count to get to: 200000 curr count: 165153
2021/12/16 23:57:29 Waiting for src bucket item count to get to: 200000 curr count: 165153
2021/12/16 23:57:34 Waiting for src bucket item count to get to: 200000 curr count: 180429
2021/12/16 23:57:39 Waiting for src bucket item count to get to: 200000 curr count: 180429
2021/12/16 23:57:44 Waiting for src bucket item count to get to: 200000 curr count: 180429
2021/12/16 23:57:49 Waiting for src bucket item count to get to: 200000 curr count: 196317
2021/12/16 23:57:54 Waiting for src bucket item count to get to: 200000 curr count: 196317
2021/12/16 23:57:59 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 23:57:59 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 23:57:59 Starting bucket ops to source bucket
2021/12/17 00:01:49 Waiting for dst bucket item count to get to: 0 curr count: 8300
2021/12/17 00:01:54 Waiting for dst bucket item count to get to: 0 curr count: 8300
2021/12/17 00:01:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:01:54 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 00:01:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:01:59 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 00:02:04 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 00:02:09 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 00:02:14 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 00:02:14 App: TestEventingSwapRebWithOps got un-deployed
2021/12/17 00:02:14 No eventing-con process running
2021/12/17 00:02:14 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/17 00:02:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:02:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:02:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:02:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:02:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:02:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (464.96s)
=== RUN   TestKVRebInWithOps
2021/12/17 00:02:38 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 00:02:38 App: TestKVRebInWithOps got un-deployed
2021/12/17 00:02:38 No eventing-con process running
2021/12/17 00:02:38 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/17 00:02:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:02:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:02:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:02:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:02:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:02:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:02:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:02:56 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:02:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 00:03:02 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 00:03:07 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 00:03:12 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 00:03:17 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 00:03:17 App: TestKVRebInWithOps got deployed
2021/12/17 00:03:17 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 00:03:17 Starting bucket ops to source bucket
2021/12/17 00:03:18 Removing node(s): [] from the cluster
2021/12/17 00:03:23 Rebalance progress: 12.5
2021/12/17 00:03:28 Rebalance progress: 12.5
2021/12/17 00:03:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:03:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:03:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:03:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:03:38 Rebalance progress: 12.5
2021/12/17 00:03:43 Rebalance progress: 12.5
2021/12/17 00:03:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:03:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:03:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:03:48 Waiting for src bucket item count to get to: 200000 curr count: 33359
2021/12/17 00:03:53 Waiting for src bucket item count to get to: 200000 curr count: 49131
2021/12/17 00:03:58 Waiting for src bucket item count to get to: 200000 curr count: 49131
2021/12/17 00:04:03 Waiting for src bucket item count to get to: 200000 curr count: 65498
2021/12/17 00:04:08 Waiting for src bucket item count to get to: 200000 curr count: 65498
2021/12/17 00:04:13 Waiting for src bucket item count to get to: 200000 curr count: 80308
2021/12/17 00:04:18 Waiting for src bucket item count to get to: 200000 curr count: 80308
2021/12/17 00:04:23 Waiting for src bucket item count to get to: 200000 curr count: 96273
2021/12/17 00:04:28 Waiting for src bucket item count to get to: 200000 curr count: 96273
2021/12/17 00:04:33 Waiting for src bucket item count to get to: 200000 curr count: 112285
2021/12/17 00:04:38 Waiting for src bucket item count to get to: 200000 curr count: 112285
2021/12/17 00:04:43 Waiting for src bucket item count to get to: 200000 curr count: 112285
2021/12/17 00:04:48 Waiting for src bucket item count to get to: 200000 curr count: 128247
2021/12/17 00:04:53 Waiting for src bucket item count to get to: 200000 curr count: 128247
2021/12/17 00:04:58 Waiting for src bucket item count to get to: 200000 curr count: 144364
2021/12/17 00:05:03 Waiting for src bucket item count to get to: 200000 curr count: 144364
2021/12/17 00:05:08 Waiting for src bucket item count to get to: 200000 curr count: 160751
2021/12/17 00:05:13 Waiting for src bucket item count to get to: 200000 curr count: 160751
2021/12/17 00:05:18 Waiting for src bucket item count to get to: 200000 curr count: 175731
2021/12/17 00:05:23 Waiting for src bucket item count to get to: 200000 curr count: 175731
2021/12/17 00:05:28 Waiting for src bucket item count to get to: 200000 curr count: 192078
2021/12/17 00:05:33 Waiting for src bucket item count to get to: 200000 curr count: 192078
2021/12/17 00:05:38 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 00:05:38 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 00:05:38 Starting bucket ops to source bucket
2021/12/17 00:09:29 Waiting for dst bucket item count to get to: 0 curr count: 7716
2021/12/17 00:09:34 Waiting for dst bucket item count to get to: 0 curr count: 7716
2021/12/17 00:09:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:09:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 00:09:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:09:39 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 00:09:44 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 00:09:49 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 00:09:54 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 00:09:54 App: TestKVRebInWithOps got un-deployed
2021/12/17 00:09:54 No eventing-con process running
2021/12/17 00:09:54 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/17 00:09:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:10:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:10:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:10:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:10:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.72s)
=== RUN   TestKVRebOutWithOps
2021/12/17 00:10:12 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 00:10:12 Removing node(s): [] from the cluster
2021/12/17 00:10:17 Rebalance progress: 100
2021/12/17 00:10:22 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 00:10:23 App: TestKVRebOutWithOps got un-deployed
2021/12/17 00:10:23 No eventing-con process running
2021/12/17 00:10:23 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/17 00:10:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:10:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:10:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:10:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:10:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:10:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:10:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:10:41 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:10:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 00:10:47 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 00:10:52 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 00:10:57 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 00:11:02 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 00:11:02 App: TestKVRebOutWithOps got deployed
2021/12/17 00:11:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:11:02 Starting bucket ops to source bucket
2021/12/17 00:11:07 Rebalance progress: 12.5
2021/12/17 00:11:12 Rebalance progress: 12.5
2021/12/17 00:11:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:11:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:11:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:11:17 Waiting for src bucket item count to get to: 200000 curr count: 10325
2021/12/17 00:11:22 Waiting for src bucket item count to get to: 200000 curr count: 26368
2021/12/17 00:11:27 Waiting for src bucket item count to get to: 200000 curr count: 26368
2021/12/17 00:11:32 Waiting for src bucket item count to get to: 200000 curr count: 42398
2021/12/17 00:11:37 Waiting for src bucket item count to get to: 200000 curr count: 42398
2021/12/17 00:11:42 Waiting for src bucket item count to get to: 200000 curr count: 58594
2021/12/17 00:11:47 Waiting for src bucket item count to get to: 200000 curr count: 58594
2021/12/17 00:11:52 Waiting for src bucket item count to get to: 200000 curr count: 58594
2021/12/17 00:11:57 Waiting for src bucket item count to get to: 200000 curr count: 73785
2021/12/17 00:12:02 Waiting for src bucket item count to get to: 200000 curr count: 73785
2021/12/17 00:12:07 Waiting for src bucket item count to get to: 200000 curr count: 89391
2021/12/17 00:12:12 Waiting for src bucket item count to get to: 200000 curr count: 89391
2021/12/17 00:12:17 Waiting for src bucket item count to get to: 200000 curr count: 104978
2021/12/17 00:12:22 Waiting for src bucket item count to get to: 200000 curr count: 104978
2021/12/17 00:12:27 Waiting for src bucket item count to get to: 200000 curr count: 121003
2021/12/17 00:12:32 Waiting for src bucket item count to get to: 200000 curr count: 121003
2021/12/17 00:12:37 Waiting for src bucket item count to get to: 200000 curr count: 137030
2021/12/17 00:12:42 Waiting for src bucket item count to get to: 200000 curr count: 137030
2021/12/17 00:12:47 Waiting for src bucket item count to get to: 200000 curr count: 152961
2021/12/17 00:12:52 Waiting for src bucket item count to get to: 200000 curr count: 152961
2021/12/17 00:12:57 Waiting for src bucket item count to get to: 200000 curr count: 168264
2021/12/17 00:13:02 Waiting for src bucket item count to get to: 200000 curr count: 168264
2021/12/17 00:13:07 Waiting for src bucket item count to get to: 200000 curr count: 184174
2021/12/17 00:13:12 Waiting for src bucket item count to get to: 200000 curr count: 184174
2021/12/17 00:13:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 00:13:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 00:13:17 Starting bucket ops to source bucket
2021/12/17 00:17:08 Waiting for dst bucket item count to get to: 0 curr count: 6805
2021/12/17 00:17:13 Waiting for dst bucket item count to get to: 0 curr count: 6805
2021/12/17 00:17:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:17:13 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 00:17:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:17:18 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 00:17:23 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 00:17:28 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 00:17:33 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 00:17:33 App: TestKVRebOutWithOps got un-deployed
2021/12/17 00:17:33 No eventing-con process running
2021/12/17 00:17:33 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/17 00:17:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:17:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:17:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:17:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:17:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:17:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (459.60s)
=== RUN   TestKVSwapRebWithOps
2021/12/17 00:17:57 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 00:17:57 App: TestKVSwapRebWithOps got un-deployed
2021/12/17 00:17:57 No eventing-con process running
2021/12/17 00:17:57 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/17 00:17:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:18:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:18:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:18:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:18:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:18:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:18:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:18:15 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:18:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 00:18:22 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 00:18:27 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 00:18:32 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 00:18:37 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 00:18:37 App: TestKVSwapRebWithOps got deployed
2021/12/17 00:18:37 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 00:18:37 Starting bucket ops to source bucket
2021/12/17 00:18:37 Removing node(s): [] from the cluster
2021/12/17 00:18:42 Rebalance progress: 12.5
2021/12/17 00:18:47 Rebalance progress: 12.5
2021/12/17 00:18:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:18:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:18:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:18:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 00:18:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:18:58 Rebalance progress: 12.5
2021/12/17 00:19:03 Rebalance progress: 12.5
2021/12/17 00:19:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:19:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:19:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:19:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 00:19:13 Rebalance progress: 12.5
2021/12/17 00:19:18 Rebalance progress: 12.5
2021/12/17 00:19:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:19:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:19:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:19:23 Waiting for src bucket item count to get to: 200000 curr count: 65037
2021/12/17 00:19:28 Waiting for src bucket item count to get to: 200000 curr count: 65037
2021/12/17 00:19:33 Waiting for src bucket item count to get to: 200000 curr count: 79620
2021/12/17 00:19:38 Waiting for src bucket item count to get to: 200000 curr count: 79620
2021/12/17 00:19:43 Waiting for src bucket item count to get to: 200000 curr count: 95796
2021/12/17 00:19:48 Waiting for src bucket item count to get to: 200000 curr count: 95796
2021/12/17 00:19:53 Waiting for src bucket item count to get to: 200000 curr count: 111797
2021/12/17 00:19:58 Waiting for src bucket item count to get to: 200000 curr count: 111797
2021/12/17 00:20:03 Waiting for src bucket item count to get to: 200000 curr count: 128063
2021/12/17 00:20:08 Waiting for src bucket item count to get to: 200000 curr count: 128063
2021/12/17 00:20:13 Waiting for src bucket item count to get to: 200000 curr count: 144079
2021/12/17 00:20:18 Waiting for src bucket item count to get to: 200000 curr count: 144079
2021/12/17 00:20:23 Waiting for src bucket item count to get to: 200000 curr count: 160098
2021/12/17 00:20:28 Waiting for src bucket item count to get to: 200000 curr count: 160098
2021/12/17 00:20:33 Waiting for src bucket item count to get to: 200000 curr count: 160098
2021/12/17 00:20:38 Waiting for src bucket item count to get to: 200000 curr count: 174921
2021/12/17 00:20:43 Waiting for src bucket item count to get to: 200000 curr count: 174921
2021/12/17 00:20:48 Waiting for src bucket item count to get to: 200000 curr count: 190947
2021/12/17 00:20:53 Waiting for src bucket item count to get to: 200000 curr count: 190947
2021/12/17 00:20:58 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 00:20:58 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 00:20:58 Starting bucket ops to source bucket
2021/12/17 00:24:48 Waiting for dst bucket item count to get to: 0 curr count: 7222
2021/12/17 00:24:53 Waiting for dst bucket item count to get to: 0 curr count: 7222
2021/12/17 00:24:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:24:53 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 00:24:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:24:58 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 00:25:03 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 00:25:08 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 00:25:13 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 00:25:13 App: TestKVSwapRebWithOps got un-deployed
2021/12/17 00:25:13 No eventing-con process running
2021/12/17 00:25:13 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/17 00:25:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:25:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:25:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:25:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:25:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:25:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (460.11s)
=== RUN   TestEventingRebInWithTimers
2021/12/17 00:25:37 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 00:25:37 App: TestEventingRebInWithTimers got un-deployed
2021/12/17 00:25:37 No eventing-con process running
2021/12/17 00:25:37 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/17 00:25:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:25:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:25:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:25:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:25:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:25:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:25:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:25:56 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:25:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 00:26:02 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 00:26:07 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 00:26:12 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 00:26:17 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 00:26:17 App: TestEventingRebInWithTimers got deployed
2021/12/17 00:26:17 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 00:26:17 Starting bucket ops to source bucket
2021/12/17 00:26:17 Removing node(s): [] from the cluster
2021/12/17 00:26:23 Rebalance progress: 12.5
2021/12/17 00:26:28 Rebalance progress: 12.5
2021/12/17 00:26:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:26:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:26:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:26:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:26:38 Rebalance progress: 12.5
2021/12/17 00:26:43 Rebalance progress: 12.5
2021/12/17 00:26:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:26:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:26:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:26:48 Waiting for src bucket item count to get to: 50000 curr count: 27573
2021/12/17 00:26:53 Waiting for src bucket item count to get to: 50000 curr count: 38132
2021/12/17 00:26:58 Waiting for src bucket item count to get to: 50000 curr count: 38132
2021/12/17 00:27:03 Waiting for src bucket item count to get to: 50000 curr count: 49191
2021/12/17 00:27:08 Waiting for src bucket item count to get to: 50000 curr count: 49191
2021/12/17 00:27:18 Waiting for dst bucket item count to get to: 50000 curr count: 44796
2021/12/17 00:27:23 Waiting for dst bucket item count to get to: 50000 curr count: 44796
2021/12/17 00:27:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 00:27:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 00:27:23 Starting bucket ops to source bucket
2021/12/17 00:28:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:28:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:28:43 Waiting for dst bucket item count to get to: 0 curr count: 49961
2021/12/17 00:28:48 Waiting for dst bucket item count to get to: 0 curr count: 49961
2021/12/17 00:28:53 Waiting for dst bucket item count to get to: 0 curr count: 41275
2021/12/17 00:28:58 Waiting for dst bucket item count to get to: 0 curr count: 41275
2021/12/17 00:29:03 Waiting for dst bucket item count to get to: 0 curr count: 31796
2021/12/17 00:29:08 Waiting for dst bucket item count to get to: 0 curr count: 31796
2021/12/17 00:29:13 Waiting for dst bucket item count to get to: 0 curr count: 8177
2021/12/17 00:29:18 Waiting for dst bucket item count to get to: 0 curr count: 8177
2021/12/17 00:29:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:29:18 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 00:29:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:29:23 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 00:29:28 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 00:29:33 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 00:29:38 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 00:29:38 App: TestEventingRebInWithTimers got un-deployed
2021/12/17 00:29:38 No eventing-con process running
2021/12/17 00:29:38 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/17 00:29:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:29:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:29:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:29:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:29:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:29:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (264.49s)
=== RUN   TestEventingRebOutWithTimers
2021/12/17 00:29:57 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 00:29:57 Removing node(s): [] from the cluster
2021/12/17 00:30:02 Rebalance progress: 100
2021/12/17 00:30:07 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 00:30:07 App: TestEventingRebOutWithTimers got un-deployed
2021/12/17 00:30:07 No eventing-con process running
2021/12/17 00:30:07 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/17 00:30:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:30:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:30:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:30:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:30:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:30:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:30:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:30:26 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:30:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 00:30:32 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 00:30:37 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 00:30:42 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 00:30:47 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 00:30:47 App: TestEventingRebOutWithTimers got deployed
2021/12/17 00:30:47 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:30:47 Starting bucket ops to source bucket
2021/12/17 00:30:52 Rebalance progress: 12.5
2021/12/17 00:30:57 Rebalance progress: 12.5
2021/12/17 00:31:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:31:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:31:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:31:02 Waiting for src bucket item count to get to: 50000 curr count: 17433
2021/12/17 00:31:07 Waiting for src bucket item count to get to: 50000 curr count: 17433
2021/12/17 00:31:12 Waiting for src bucket item count to get to: 50000 curr count: 27345
2021/12/17 00:31:17 Waiting for src bucket item count to get to: 50000 curr count: 27345
2021/12/17 00:31:22 Waiting for src bucket item count to get to: 50000 curr count: 38690
2021/12/17 00:31:27 Waiting for src bucket item count to get to: 50000 curr count: 38690
2021/12/17 00:31:32 Waiting for src bucket item count to get to: 50000 curr count: 38690
2021/12/17 00:31:42 Waiting for dst bucket item count to get to: 50000 curr count: 37382
2021/12/17 00:31:47 Waiting for dst bucket item count to get to: 50000 curr count: 37382
2021/12/17 00:31:52 Waiting for dst bucket item count to get to: 50000 curr count: 49242
2021/12/17 00:31:57 Waiting for dst bucket item count to get to: 50000 curr count: 49242
2021/12/17 00:31:57 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 00:31:57 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 00:31:57 Starting bucket ops to source bucket
2021/12/17 00:33:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:33:12 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:33:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:33:22 Waiting for dst bucket item count to get to: 0 curr count: 43193
2021/12/17 00:33:27 Waiting for dst bucket item count to get to: 0 curr count: 43193
2021/12/17 00:33:32 Waiting for dst bucket item count to get to: 0 curr count: 31946
2021/12/17 00:33:37 Waiting for dst bucket item count to get to: 0 curr count: 31946
2021/12/17 00:33:42 Waiting for dst bucket item count to get to: 0 curr count: 10836
2021/12/17 00:33:47 Waiting for dst bucket item count to get to: 0 curr count: 10836
2021/12/17 00:33:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:33:47 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 00:33:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:33:52 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 00:33:58 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 00:34:03 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 00:34:08 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 00:34:08 App: TestEventingRebOutWithTimers got un-deployed
2021/12/17 00:34:08 No eventing-con process running
2021/12/17 00:34:08 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/17 00:34:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:34:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:34:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:34:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:34:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:34:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (269.58s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/17 00:34:31 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 00:34:31 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/17 00:34:31 No eventing-con process running
2021/12/17 00:34:31 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/17 00:34:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:34:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:34:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:34:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:34:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:34:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:34:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:34:50 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:34:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 00:34:56 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 00:35:01 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 00:35:06 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 00:35:11 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 00:35:11 App: TestEventingSwapRebWithTimers got deployed
2021/12/17 00:35:11 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 00:35:11 Starting bucket ops to source bucket
2021/12/17 00:35:11 Removing node(s): [] from the cluster
2021/12/17 00:35:16 Rebalance progress: 12.5
2021/12/17 00:35:21 Rebalance progress: 12.5
2021/12/17 00:35:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:35:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:35:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:35:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 00:35:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:35:32 Rebalance progress: 12.5
2021/12/17 00:35:37 Rebalance progress: 12.5
2021/12/17 00:35:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:35:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:35:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:35:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 00:35:47 Rebalance progress: 12.5
2021/12/17 00:35:52 Rebalance progress: 12.5
2021/12/17 00:35:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:35:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:35:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:35:57 Waiting for src bucket item count to get to: 50000 curr count: 44477
2021/12/17 00:36:07 Waiting for dst bucket item count to get to: 50000 curr count: 34713
2021/12/17 00:36:12 Waiting for dst bucket item count to get to: 50000 curr count: 34713
2021/12/17 00:36:17 Waiting for dst bucket item count to get to: 50000 curr count: 49721
2021/12/17 00:36:22 Waiting for dst bucket item count to get to: 50000 curr count: 49721
2021/12/17 00:36:22 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 00:36:22 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 00:36:22 Starting bucket ops to source bucket
2021/12/17 00:37:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:37:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:37:42 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:37:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:37:52 Waiting for dst bucket item count to get to: 0 curr count: 40070
2021/12/17 00:37:57 Waiting for dst bucket item count to get to: 0 curr count: 40070
2021/12/17 00:38:02 Waiting for dst bucket item count to get to: 0 curr count: 29035
2021/12/17 00:38:07 Waiting for dst bucket item count to get to: 0 curr count: 29035
2021/12/17 00:38:12 Waiting for dst bucket item count to get to: 0 curr count: 13270
2021/12/17 00:38:17 Waiting for dst bucket item count to get to: 0 curr count: 13270
2021/12/17 00:38:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:38:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 00:38:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:38:22 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 00:38:27 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 00:38:32 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 00:38:37 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 00:38:37 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/17 00:38:37 No eventing-con process running
2021/12/17 00:38:37 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/17 00:38:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:38:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:38:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:38:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:38:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:38:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (269.65s)
=== RUN   TestKVRebInWithTimers
2021/12/17 00:39:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 00:39:01 App: TestKVRebInWithTimers got un-deployed
2021/12/17 00:39:01 No eventing-con process running
2021/12/17 00:39:01 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/17 00:39:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:39:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:39:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:39:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:39:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:39:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:39:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:39:19 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:39:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 00:39:25 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 00:39:30 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 00:39:35 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 00:39:40 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 00:39:40 App: TestKVRebInWithTimers got deployed
2021/12/17 00:39:40 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 00:39:40 Starting bucket ops to source bucket
2021/12/17 00:39:42 Removing node(s): [] from the cluster
2021/12/17 00:39:47 Rebalance progress: 12.5
2021/12/17 00:39:52 Rebalance progress: 12.5
2021/12/17 00:39:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:39:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:39:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:39:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:40:02 Rebalance progress: 12.5
2021/12/17 00:40:07 Rebalance progress: 12.5
2021/12/17 00:40:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:40:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:40:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:40:12 Waiting for src bucket item count to get to: 50000 curr count: 33821
2021/12/17 00:40:17 Waiting for src bucket item count to get to: 50000 curr count: 33821
2021/12/17 00:40:22 Waiting for src bucket item count to get to: 50000 curr count: 45316
2021/12/17 00:40:28 Waiting for src bucket item count to get to: 50000 curr count: 45316
2021/12/17 00:40:38 Waiting for dst bucket item count to get to: 50000 curr count: 41051
2021/12/17 00:40:43 Waiting for dst bucket item count to get to: 50000 curr count: 41051
2021/12/17 00:40:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 00:40:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 00:40:43 Starting bucket ops to source bucket
2021/12/17 00:41:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:41:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:42:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:42:08 Waiting for dst bucket item count to get to: 0 curr count: 44330
2021/12/17 00:42:13 Waiting for dst bucket item count to get to: 0 curr count: 44330
2021/12/17 00:42:18 Waiting for dst bucket item count to get to: 0 curr count: 26384
2021/12/17 00:42:23 Waiting for dst bucket item count to get to: 0 curr count: 26384
2021/12/17 00:42:28 Waiting for dst bucket item count to get to: 0 curr count: 9704
2021/12/17 00:42:33 Waiting for dst bucket item count to get to: 0 curr count: 9704
2021/12/17 00:42:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:42:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 00:42:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:42:38 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 00:42:43 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 00:42:49 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 00:42:54 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 00:42:54 App: TestKVRebInWithTimers got un-deployed
2021/12/17 00:42:54 No eventing-con process running
2021/12/17 00:42:54 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/17 00:42:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:43:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:43:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:43:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:43:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:43:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (256.36s)
=== RUN   TestKVRebOutWithTimers
2021/12/17 00:43:12 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 00:43:13 Removing node(s): [] from the cluster
2021/12/17 00:43:18 Rebalance progress: 100
2021/12/17 00:43:23 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 00:43:23 App: TestKVRebOutWithTimers got un-deployed
2021/12/17 00:43:23 No eventing-con process running
2021/12/17 00:43:23 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/17 00:43:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:43:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:43:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:43:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:43:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:43:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:43:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:43:41 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:43:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 00:43:47 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 00:43:52 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 00:43:57 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 00:44:02 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 00:44:02 App: TestKVRebOutWithTimers got deployed
2021/12/17 00:44:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:44:02 Starting bucket ops to source bucket
2021/12/17 00:44:07 Rebalance progress: 12.5
2021/12/17 00:44:12 Rebalance progress: 12.5
2021/12/17 00:44:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:44:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:44:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:44:17 Waiting for src bucket item count to get to: 50000 curr count: 9265
2021/12/17 00:44:22 Waiting for src bucket item count to get to: 50000 curr count: 23251
2021/12/17 00:44:27 Waiting for src bucket item count to get to: 50000 curr count: 23251
2021/12/17 00:44:32 Waiting for src bucket item count to get to: 50000 curr count: 34728
2021/12/17 00:44:37 Waiting for src bucket item count to get to: 50000 curr count: 34728
2021/12/17 00:44:43 Waiting for src bucket item count to get to: 50000 curr count: 34728
2021/12/17 00:44:48 Waiting for src bucket item count to get to: 50000 curr count: 44988
2021/12/17 00:44:53 Waiting for src bucket item count to get to: 50000 curr count: 44988
2021/12/17 00:45:03 Waiting for dst bucket item count to get to: 50000 curr count: 35140
2021/12/17 00:45:08 Waiting for dst bucket item count to get to: 50000 curr count: 35140
2021/12/17 00:45:08 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 00:45:08 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 00:45:08 Starting bucket ops to source bucket
2021/12/17 00:46:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:46:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:46:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:46:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:46:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:46:43 Waiting for dst bucket item count to get to: 0 curr count: 37530
2021/12/17 00:46:48 Waiting for dst bucket item count to get to: 0 curr count: 37530
2021/12/17 00:46:53 Waiting for dst bucket item count to get to: 0 curr count: 19562
2021/12/17 00:46:58 Waiting for dst bucket item count to get to: 0 curr count: 7666
2021/12/17 00:47:03 Waiting for dst bucket item count to get to: 0 curr count: 7666
2021/12/17 00:47:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:47:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 00:47:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:47:08 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 00:47:13 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 00:47:18 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 00:47:23 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 00:47:23 App: TestKVRebOutWithTimers got un-deployed
2021/12/17 00:47:23 No eventing-con process running
2021/12/17 00:47:23 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/17 00:47:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:47:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:47:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:47:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:47:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:47:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (269.81s)
=== RUN   TestKVSwapRebWithTimers
2021/12/17 00:47:47 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 00:47:47 App: TestKVSwapRebWithTimers got un-deployed
2021/12/17 00:47:47 No eventing-con process running
2021/12/17 00:47:47 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/17 00:47:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:47:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:47:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:47:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:48:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:48:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:48:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:48:06 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:48:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 00:48:12 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 00:48:17 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 00:48:22 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 00:48:27 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 00:48:27 App: TestKVSwapRebWithTimers got deployed
2021/12/17 00:48:27 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 00:48:27 Starting bucket ops to source bucket
2021/12/17 00:48:27 Removing node(s): [] from the cluster
2021/12/17 00:48:32 Rebalance progress: 12.5
2021/12/17 00:48:37 Rebalance progress: 12.5
2021/12/17 00:48:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:48:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:48:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:48:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 00:48:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:48:48 Rebalance progress: 12.5
2021/12/17 00:48:53 Rebalance progress: 12.5
2021/12/17 00:48:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:48:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:48:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:48:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 00:49:03 Rebalance progress: 12.5
2021/12/17 00:49:08 Rebalance progress: 12.5
2021/12/17 00:49:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:49:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:49:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:49:13 Waiting for src bucket item count to get to: 50000 curr count: 49099
2021/12/17 00:49:18 Waiting for src bucket item count to get to: 50000 curr count: 49099
2021/12/17 00:49:28 Waiting for dst bucket item count to get to: 50000 curr count: 44573
2021/12/17 00:49:33 Waiting for dst bucket item count to get to: 50000 curr count: 44573
2021/12/17 00:49:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 00:49:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 00:49:33 Starting bucket ops to source bucket
2021/12/17 00:50:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:50:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:50:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:50:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:51:03 Waiting for dst bucket item count to get to: 0 curr count: 41793
2021/12/17 00:51:08 Waiting for dst bucket item count to get to: 0 curr count: 41793
2021/12/17 00:51:13 Waiting for dst bucket item count to get to: 0 curr count: 32545
2021/12/17 00:51:18 Waiting for dst bucket item count to get to: 0 curr count: 32545
2021/12/17 00:51:23 Waiting for dst bucket item count to get to: 0 curr count: 9313
2021/12/17 00:51:28 Waiting for dst bucket item count to get to: 0 curr count: 9313
2021/12/17 00:51:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:51:28 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 00:51:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:51:33 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 00:51:38 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 00:51:43 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 00:51:48 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 00:51:54 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 00:51:54 App: TestKVSwapRebWithTimers got un-deployed
2021/12/17 00:51:54 No eventing-con process running
2021/12/17 00:51:54 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/17 00:51:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:52:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:52:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:52:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:52:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:52:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (270.29s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/17 00:52:17 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 00:52:17 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/17 00:52:17 No eventing-con process running
2021/12/17 00:52:17 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/17 00:52:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:52:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:52:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:52:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:52:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:52:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:52:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:52:36 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:52:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 00:52:42 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 00:52:47 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 00:52:52 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 00:52:57 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 00:52:57 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/17 00:52:57 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 00:52:57 Starting bucket ops to source bucket
2021/12/17 00:52:58 Removing node(s): [] from the cluster
2021/12/17 00:53:03 Rebalance progress: 12.5
2021/12/17 00:53:08 Rebalance progress: 12.5
2021/12/17 00:53:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:53:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:53:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:53:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:53:18 Rebalance progress: 12.5
2021/12/17 00:53:23 Rebalance progress: 12.5
2021/12/17 00:53:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:53:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:53:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:53:28 Waiting for src bucket item count to get to: 50000 curr count: 27641
2021/12/17 00:53:33 Waiting for src bucket item count to get to: 50000 curr count: 38361
2021/12/17 00:53:38 Waiting for src bucket item count to get to: 50000 curr count: 38361
2021/12/17 00:53:43 Waiting for src bucket item count to get to: 50000 curr count: 48941
2021/12/17 00:53:48 Waiting for src bucket item count to get to: 50000 curr count: 48941
2021/12/17 00:53:58 Waiting for dst bucket item count to get to: 50000 curr count: 40037
2021/12/17 00:54:03 Waiting for dst bucket item count to get to: 50000 curr count: 40037
2021/12/17 00:54:08 Waiting for dst bucket item count to get to: 50000 curr count: 47622
2021/12/17 00:54:13 Waiting for dst bucket item count to get to: 50000 curr count: 47622
2021/12/17 00:54:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 00:54:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 00:54:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:54:18 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 00:54:23 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 00:54:28 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 00:54:33 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 00:54:38 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 00:54:43 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 00:54:43 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/17 00:54:43 No eventing-con process running
2021/12/17 00:54:43 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/17 00:54:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:54:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:54:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:54:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:54:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:55:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:55:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:55:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.58s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/17 00:55:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 00:55:12 Removing node(s): [] from the cluster
2021/12/17 00:55:17 Rebalance progress: 100
2021/12/17 00:55:22 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 00:55:22 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/17 00:55:22 No eventing-con process running
2021/12/17 00:55:22 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/17 00:55:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:55:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:55:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:55:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:55:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:55:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:55:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:55:41 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:55:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 00:55:47 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 00:55:52 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 00:55:57 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 00:56:02 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 00:56:02 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/17 00:56:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:56:02 Starting bucket ops to source bucket
2021/12/17 00:56:07 Rebalance progress: 12.5
2021/12/17 00:56:12 Rebalance progress: 12.5
2021/12/17 00:56:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:56:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:56:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:56:17 Waiting for src bucket item count to get to: 50000 curr count: 9273
2021/12/17 00:56:22 Waiting for src bucket item count to get to: 50000 curr count: 22600
2021/12/17 00:56:27 Waiting for src bucket item count to get to: 50000 curr count: 22600
2021/12/17 00:56:32 Waiting for src bucket item count to get to: 50000 curr count: 35735
2021/12/17 00:56:37 Waiting for src bucket item count to get to: 50000 curr count: 35735
2021/12/17 00:56:42 Waiting for src bucket item count to get to: 50000 curr count: 45048
2021/12/17 00:56:47 Waiting for src bucket item count to get to: 50000 curr count: 45048
2021/12/17 00:56:52 Waiting for src bucket item count to get to: 50000 curr count: 45048
2021/12/17 00:57:02 Waiting for dst bucket item count to get to: 50000 curr count: 30350
2021/12/17 00:57:07 Waiting for dst bucket item count to get to: 50000 curr count: 30350
2021/12/17 00:57:12 Waiting for dst bucket item count to get to: 50000 curr count: 43727
2021/12/17 00:57:17 Waiting for dst bucket item count to get to: 50000 curr count: 43727
2021/12/17 00:57:17 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 00:57:17 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 00:57:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 00:57:22 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 00:57:27 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 00:57:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 00:57:37 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 00:57:42 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 00:57:47 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 00:57:47 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/17 00:57:47 No eventing-con process running
2021/12/17 00:57:48 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/17 00:57:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:57:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:57:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:57:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:58:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:58:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:58:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:58:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 00:58:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.34s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/17 00:58:26 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 00:58:26 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/17 00:58:26 No eventing-con process running
2021/12/17 00:58:26 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/17 00:58:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:58:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:58:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 00:58:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:58:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:58:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:58:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 00:58:45 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 00:58:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 00:58:51 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 00:58:56 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 00:59:01 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 00:59:06 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 00:59:06 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/17 00:59:06 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 00:59:06 Starting bucket ops to source bucket
2021/12/17 00:59:06 Removing node(s): [] from the cluster
2021/12/17 00:59:11 Rebalance progress: 12.5
2021/12/17 00:59:16 Rebalance progress: 12.5
2021/12/17 00:59:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:59:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:59:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:59:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 00:59:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 00:59:27 Rebalance progress: 12.5
2021/12/17 00:59:32 Rebalance progress: 12.5
2021/12/17 00:59:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:59:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:59:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:59:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 00:59:42 Rebalance progress: 12.5
2021/12/17 00:59:47 Rebalance progress: 12.5
2021/12/17 00:59:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:59:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 00:59:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 00:59:57 Waiting for dst bucket item count to get to: 50000 curr count: 27112
2021/12/17 01:00:02 Waiting for dst bucket item count to get to: 50000 curr count: 27112
2021/12/17 01:00:07 Waiting for dst bucket item count to get to: 50000 curr count: 36100
2021/12/17 01:00:12 Waiting for dst bucket item count to get to: 50000 curr count: 36100
2021/12/17 01:00:17 Waiting for dst bucket item count to get to: 50000 curr count: 46567
2021/12/17 01:00:22 Waiting for dst bucket item count to get to: 50000 curr count: 46567
2021/12/17 01:00:22 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 01:00:22 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 01:00:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:00:27 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 01:00:32 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 01:00:37 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 01:00:42 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 01:00:47 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 01:00:52 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 01:00:52 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/17 01:00:52 No eventing-con process running
2021/12/17 01:00:52 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/17 01:00:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:00:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:01:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:01:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:01:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:01:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:01:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:01:21 Waiting for dst bucket item count to get to: 0 curr count: 44245
2021/12/17 01:01:26 Waiting for dst bucket item count to get to: 0 curr count: 44245
2021/12/17 01:01:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (189.76s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/17 01:01:36 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 01:01:36 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/17 01:01:36 No eventing-con process running
2021/12/17 01:01:36 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/17 01:01:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:01:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:01:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:01:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:01:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:01:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:01:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:01:54 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:01:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 01:02:01 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 01:02:06 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 01:02:11 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 01:02:16 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 01:02:16 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/17 01:02:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:02:16 Starting bucket ops to source bucket
2021/12/17 01:02:16 Removing node(s): [] from the cluster
2021/12/17 01:02:21 Rebalance progress: 12.5
2021/12/17 01:02:26 Rebalance progress: 12.5
2021/12/17 01:02:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:02:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:02:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:02:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:02:36 Rebalance progress: 12.5
2021/12/17 01:02:41 Rebalance progress: 12.5
2021/12/17 01:02:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:02:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:02:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:02:46 Waiting for src bucket item count to get to: 50000 curr count: 29553
2021/12/17 01:02:51 Waiting for src bucket item count to get to: 50000 curr count: 40787
2021/12/17 01:02:56 Waiting for src bucket item count to get to: 50000 curr count: 40787
2021/12/17 01:03:06 Waiting for dst bucket item count to get to: 50000 curr count: 24865
2021/12/17 01:03:11 Waiting for dst bucket item count to get to: 50000 curr count: 24865
2021/12/17 01:03:16 Waiting for dst bucket item count to get to: 50000 curr count: 34574
2021/12/17 01:03:21 Waiting for dst bucket item count to get to: 50000 curr count: 34574
2021/12/17 01:03:26 Waiting for dst bucket item count to get to: 50000 curr count: 47090
2021/12/17 01:03:31 Waiting for dst bucket item count to get to: 50000 curr count: 47090
2021/12/17 01:03:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 01:03:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 01:03:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:03:36 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 01:03:41 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 01:03:47 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 01:03:52 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 01:03:57 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 01:04:02 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 01:04:02 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/17 01:04:02 No eventing-con process running
2021/12/17 01:04:02 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/17 01:04:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:04:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:04:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:04:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:04:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:04:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:04:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:04:30 Waiting for dst bucket item count to get to: 0 curr count: 15264
2021/12/17 01:04:35 Waiting for dst bucket item count to get to: 0 curr count: 15264
2021/12/17 01:04:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (189.33s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/17 01:04:40 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:04:40 Removing node(s): [] from the cluster
2021/12/17 01:04:45 Rebalance progress: 100
2021/12/17 01:04:50 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 01:04:50 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/17 01:04:50 No eventing-con process running
2021/12/17 01:04:50 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/17 01:04:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:04:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:04:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:04:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:05:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:05:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:05:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:05:09 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:05:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 01:05:15 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 01:05:20 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 01:05:25 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 01:05:30 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 01:05:30 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/17 01:05:30 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:05:30 Starting bucket ops to source bucket
2021/12/17 01:05:35 Rebalance progress: 12.5
2021/12/17 01:05:40 Rebalance progress: 12.5
2021/12/17 01:05:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:05:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:05:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:05:45 Waiting for src bucket item count to get to: 50000 curr count: 11539
2021/12/17 01:05:50 Waiting for src bucket item count to get to: 50000 curr count: 25071
2021/12/17 01:05:55 Waiting for src bucket item count to get to: 50000 curr count: 25071
2021/12/17 01:06:00 Waiting for src bucket item count to get to: 50000 curr count: 36807
2021/12/17 01:06:05 Waiting for src bucket item count to get to: 50000 curr count: 36807
2021/12/17 01:06:10 Waiting for src bucket item count to get to: 50000 curr count: 36807
2021/12/17 01:06:15 Waiting for src bucket item count to get to: 50000 curr count: 46275
2021/12/17 01:06:20 Waiting for src bucket item count to get to: 50000 curr count: 46275
2021/12/17 01:06:30 Waiting for dst bucket item count to get to: 50000 curr count: 32233
2021/12/17 01:06:35 Waiting for dst bucket item count to get to: 50000 curr count: 32233
2021/12/17 01:06:40 Waiting for dst bucket item count to get to: 50000 curr count: 42326
2021/12/17 01:06:45 Waiting for dst bucket item count to get to: 50000 curr count: 42326
2021/12/17 01:06:50 Waiting for dst bucket item count to get to: 50000 curr count: 49255
2021/12/17 01:06:55 Waiting for dst bucket item count to get to: 50000 curr count: 49255
2021/12/17 01:06:55 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 01:06:55 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 01:06:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:07:01 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 01:07:06 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 01:07:11 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 01:07:16 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 01:07:21 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 01:07:26 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 01:07:26 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/17 01:07:26 No eventing-con process running
2021/12/17 01:07:26 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/17 01:07:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:07:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:07:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:07:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:07:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:07:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:07:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.10s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/17 01:07:54 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 01:07:54 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/17 01:07:54 No eventing-con process running
2021/12/17 01:07:54 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/17 01:07:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:08:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:08:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:08:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:08:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:08:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:08:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:08:13 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:08:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 01:08:19 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 01:08:24 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 01:08:29 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 01:08:34 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 01:08:34 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/17 01:08:34 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:08:34 Starting bucket ops to source bucket
2021/12/17 01:08:34 Removing node(s): [] from the cluster
2021/12/17 01:08:40 Rebalance progress: 12.5
2021/12/17 01:08:45 Rebalance progress: 12.5
2021/12/17 01:08:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:08:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:08:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:08:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 01:08:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:08:55 Rebalance progress: 12.5
2021/12/17 01:09:00 Rebalance progress: 12.5
2021/12/17 01:09:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:09:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:09:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:09:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 01:09:10 Rebalance progress: 12.5
2021/12/17 01:09:15 Rebalance progress: 12.5
2021/12/17 01:09:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:09:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:09:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:09:21 Waiting for src bucket item count to get to: 50000 curr count: 41981
2021/12/17 01:09:31 Waiting for dst bucket item count to get to: 50000 curr count: 27960
2021/12/17 01:09:36 Waiting for dst bucket item count to get to: 50000 curr count: 27960
2021/12/17 01:09:41 Waiting for dst bucket item count to get to: 50000 curr count: 36461
2021/12/17 01:09:46 Waiting for dst bucket item count to get to: 50000 curr count: 36461
2021/12/17 01:09:51 Waiting for dst bucket item count to get to: 50000 curr count: 48736
2021/12/17 01:09:51 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 01:09:51 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 01:09:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:09:56 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 01:10:01 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 01:10:06 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 01:10:11 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 01:10:16 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 01:10:21 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 01:10:21 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/17 01:10:21 No eventing-con process running
2021/12/17 01:10:21 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/17 01:10:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:10:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:10:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:10:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:10:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:10:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:10:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:10:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (180.14s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/17 01:10:54 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 01:10:54 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/17 01:10:54 No eventing-con process running
2021/12/17 01:10:55 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/17 01:10:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:11:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:11:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:11:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:11:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:11:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:11:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:11:13 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:11:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 01:11:19 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 01:11:24 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 01:11:29 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 01:11:34 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 01:11:34 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/17 01:11:34 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 01:11:34 Starting bucket ops to source bucket
2021/12/17 01:11:35 Removing node(s): [] from the cluster
2021/12/17 01:11:40 Rebalance progress: 12.5
2021/12/17 01:11:45 Rebalance progress: 12.5
2021/12/17 01:11:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:11:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:11:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:11:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:11:55 Rebalance progress: 12.5
2021/12/17 01:12:00 Rebalance progress: 12.5
2021/12/17 01:12:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:12:05 Waiting for src bucket item count to get to: 50000 curr count: 29754
2021/12/17 01:12:10 Waiting for src bucket item count to get to: 50000 curr count: 29754
2021/12/17 01:12:15 Waiting for src bucket item count to get to: 50000 curr count: 41124
2021/12/17 01:12:20 Waiting for src bucket item count to get to: 50000 curr count: 41124
2021/12/17 01:12:30 Waiting for dst bucket item count to get to: 50000 curr count: 33267
2021/12/17 01:12:35 Waiting for dst bucket item count to get to: 50000 curr count: 43245
2021/12/17 01:12:40 Waiting for dst bucket item count to get to: 50000 curr count: 43245
2021/12/17 01:12:40 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 01:12:40 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 01:12:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:12:45 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 01:12:50 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 01:12:55 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 01:13:00 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 01:13:05 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 01:13:10 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 01:13:11 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/17 01:13:11 No eventing-con process running
2021/12/17 01:13:11 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/17 01:13:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:13:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:13:22 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:13:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:13:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:13:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:13:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:13:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:13:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (174.69s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/17 01:13:44 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 01:13:44 Removing node(s): [] from the cluster
2021/12/17 01:13:50 Rebalance progress: 100
2021/12/17 01:13:55 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 01:13:55 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/17 01:13:55 No eventing-con process running
2021/12/17 01:13:55 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/17 01:13:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:14:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:14:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:14:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:14:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:14:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:14:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:14:13 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:14:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 01:14:19 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 01:14:24 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 01:14:29 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 01:14:34 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 01:14:34 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/17 01:14:34 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:14:34 Starting bucket ops to source bucket
2021/12/17 01:14:39 Rebalance progress: 12.5
2021/12/17 01:14:44 Rebalance progress: 12.5
2021/12/17 01:14:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:14:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:14:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:14:49 Waiting for src bucket item count to get to: 50000 curr count: 6382
2021/12/17 01:14:54 Waiting for src bucket item count to get to: 50000 curr count: 19698
2021/12/17 01:14:59 Waiting for src bucket item count to get to: 50000 curr count: 19698
2021/12/17 01:15:05 Waiting for src bucket item count to get to: 50000 curr count: 30665
2021/12/17 01:15:10 Waiting for src bucket item count to get to: 50000 curr count: 30665
2021/12/17 01:15:15 Waiting for src bucket item count to get to: 50000 curr count: 30665
2021/12/17 01:15:20 Waiting for src bucket item count to get to: 50000 curr count: 42350
2021/12/17 01:15:30 Waiting for dst bucket item count to get to: 50000 curr count: 38721
2021/12/17 01:15:35 Waiting for dst bucket item count to get to: 50000 curr count: 38721
2021/12/17 01:15:40 Waiting for dst bucket item count to get to: 50000 curr count: 46945
2021/12/17 01:15:45 Waiting for dst bucket item count to get to: 50000 curr count: 46945
2021/12/17 01:15:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 01:15:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 01:15:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:15:50 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 01:15:55 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 01:16:00 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 01:16:05 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 01:16:10 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 01:16:15 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 01:16:15 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/17 01:16:15 No eventing-con process running
2021/12/17 01:16:15 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/17 01:16:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:16:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:16:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:16:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:16:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:16:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:16:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:16:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.33s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/17 01:16:49 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 01:16:49 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/17 01:16:49 No eventing-con process running
2021/12/17 01:16:49 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/17 01:16:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:16:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:16:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:16:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:17:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:17:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:17:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:17:07 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:17:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 01:17:13 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 01:17:18 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 01:17:23 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 01:17:28 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 01:17:28 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/17 01:17:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 01:17:28 Starting bucket ops to source bucket
2021/12/17 01:17:28 Removing node(s): [] from the cluster
2021/12/17 01:17:34 Rebalance progress: 12.5
2021/12/17 01:17:39 Rebalance progress: 12.5
2021/12/17 01:17:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:17:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:17:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:17:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 01:17:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:17:49 Rebalance progress: 12.5
2021/12/17 01:17:54 Rebalance progress: 12.5
2021/12/17 01:17:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:17:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:17:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:17:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 01:18:04 Rebalance progress: 12.5
2021/12/17 01:18:09 Rebalance progress: 12.5
2021/12/17 01:18:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:18:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:18:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:18:14 Waiting for src bucket item count to get to: 50000 curr count: 46746
2021/12/17 01:18:19 Waiting for src bucket item count to get to: 50000 curr count: 46746
2021/12/17 01:18:29 Waiting for dst bucket item count to get to: 50000 curr count: 45576
2021/12/17 01:18:34 Waiting for dst bucket item count to get to: 50000 curr count: 45576
2021/12/17 01:18:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 01:18:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 01:18:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:18:39 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 01:18:44 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 01:18:49 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 01:18:54 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 01:18:59 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 01:19:04 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 01:19:04 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/17 01:19:04 No eventing-con process running
2021/12/17 01:19:04 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/17 01:19:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:19:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:19:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:19:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:19:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:19:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:19:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:19:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (169.56s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/17 01:19:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 01:19:38 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/17 01:19:38 No eventing-con process running
2021/12/17 01:19:38 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/17 01:19:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:19:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:19:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:19:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:19:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:19:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:19:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:19:57 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:19:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 01:20:03 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 01:20:08 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 01:20:13 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 01:20:18 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 01:20:18 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/17 01:20:18 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:20:18 Starting bucket ops to source bucket
2021/12/17 01:20:19 Removing node(s): [] from the cluster
2021/12/17 01:20:24 Rebalance progress: 12.5
2021/12/17 01:20:29 Rebalance progress: 12.5
2021/12/17 01:20:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:20:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:20:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:20:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:20:39 Rebalance progress: 12.5
2021/12/17 01:20:44 Rebalance progress: 12.5
2021/12/17 01:20:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:20:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:20:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:20:49 Waiting for src bucket item count to get to: 50000 curr count: 26399
2021/12/17 01:20:54 Waiting for src bucket item count to get to: 50000 curr count: 36157
2021/12/17 01:20:59 Waiting for src bucket item count to get to: 50000 curr count: 36157
2021/12/17 01:21:04 Waiting for src bucket item count to get to: 50000 curr count: 48077
2021/12/17 01:21:09 Waiting for src bucket item count to get to: 50000 curr count: 48077
2021/12/17 01:21:19 Waiting for dst bucket item count to get to: 50000 curr count: 44294
2021/12/17 01:21:24 Waiting for dst bucket item count to get to: 50000 curr count: 44294
2021/12/17 01:21:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 01:21:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 01:21:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:21:29 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 01:21:34 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 01:21:39 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 01:21:44 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 01:21:49 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 01:21:54 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 01:21:55 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/17 01:21:55 No eventing-con process running
2021/12/17 01:21:55 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/17 01:21:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:22:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:22:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:22:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:22:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:22:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:22:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:22:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (170.02s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/17 01:22:23 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:22:24 Removing node(s): [] from the cluster
2021/12/17 01:22:29 Rebalance progress: 100
2021/12/17 01:22:34 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 01:22:34 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/17 01:22:34 No eventing-con process running
2021/12/17 01:22:34 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/17 01:22:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:22:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:22:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:22:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:22:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:22:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:22:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:22:52 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:22:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 01:22:58 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 01:23:03 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 01:23:08 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 01:23:13 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 01:23:13 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/17 01:23:13 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:23:13 Starting bucket ops to source bucket
2021/12/17 01:23:18 Rebalance progress: 12.5
2021/12/17 01:23:23 Rebalance progress: 12.5
2021/12/17 01:23:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:23:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:23:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:23:28 Waiting for src bucket item count to get to: 50000 curr count: 7750
2021/12/17 01:23:33 Waiting for src bucket item count to get to: 50000 curr count: 20866
2021/12/17 01:23:38 Waiting for src bucket item count to get to: 50000 curr count: 20866
2021/12/17 01:23:43 Waiting for src bucket item count to get to: 50000 curr count: 32211
2021/12/17 01:23:48 Waiting for src bucket item count to get to: 50000 curr count: 32211
2021/12/17 01:23:53 Waiting for src bucket item count to get to: 50000 curr count: 32211
2021/12/17 01:23:58 Waiting for src bucket item count to get to: 50000 curr count: 42720
2021/12/17 01:24:03 Waiting for src bucket item count to get to: 50000 curr count: 42720
2021/12/17 01:24:13 Waiting for dst bucket item count to get to: 50000 curr count: 37050
2021/12/17 01:24:18 Waiting for dst bucket item count to get to: 50000 curr count: 37050
2021/12/17 01:24:23 Waiting for dst bucket item count to get to: 50000 curr count: 47963
2021/12/17 01:24:28 Waiting for dst bucket item count to get to: 50000 curr count: 47963
2021/12/17 01:24:28 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 01:24:28 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 01:24:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:24:34 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 01:24:39 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 01:24:44 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 01:24:49 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 01:24:54 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 01:24:59 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 01:24:59 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/17 01:24:59 No eventing-con process running
2021/12/17 01:24:59 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/17 01:25:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:25:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:25:11 Waiting for dst bucket item count to get to: 0 curr count: 39509
2021/12/17 01:25:16 Waiting for dst bucket item count to get to: 0 curr count: 39509
2021/12/17 01:25:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:25:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:25:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:25:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (189.52s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/17 01:25:38 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 01:25:38 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/17 01:25:38 No eventing-con process running
2021/12/17 01:25:38 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/17 01:25:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:25:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:25:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:25:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:25:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:25:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:25:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:25:56 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:25:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 01:26:02 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 01:26:07 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 01:26:12 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 01:26:17 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 01:26:17 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/17 01:26:17 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:26:17 Starting bucket ops to source bucket
2021/12/17 01:26:18 Removing node(s): [] from the cluster
2021/12/17 01:26:23 Rebalance progress: 12.5
2021/12/17 01:26:28 Rebalance progress: 12.5
2021/12/17 01:26:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:26:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:26:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:26:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 01:26:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:26:38 Rebalance progress: 12.5
2021/12/17 01:26:43 Rebalance progress: 12.5
2021/12/17 01:26:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:26:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:26:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:26:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 01:26:53 Rebalance progress: 12.5
2021/12/17 01:26:58 Rebalance progress: 12.5
2021/12/17 01:27:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:27:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:27:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:27:04 Waiting for src bucket item count to get to: 50000 curr count: 48260
2021/12/17 01:27:09 Waiting for src bucket item count to get to: 50000 curr count: 48260
2021/12/17 01:27:19 Waiting for dst bucket item count to get to: 50000 curr count: 39655
2021/12/17 01:27:24 Waiting for dst bucket item count to get to: 50000 curr count: 39655
2021/12/17 01:27:29 Waiting for dst bucket item count to get to: 50000 curr count: 47764
2021/12/17 01:27:34 Waiting for dst bucket item count to get to: 50000 curr count: 47764
2021/12/17 01:27:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 01:27:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 01:27:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:27:39 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 01:27:44 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 01:27:49 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 01:27:54 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 01:27:59 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 01:28:04 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 01:28:04 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/17 01:28:04 No eventing-con process running
2021/12/17 01:28:04 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/17 01:28:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:28:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:28:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:28:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:28:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:28:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:28:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 01:28:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (179.96s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/17 01:28:38 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:28:38 App: bucket_op_on_update got un-deployed
2021/12/17 01:28:38 No eventing-con process running
2021/12/17 01:28:38 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 01:28:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:28:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:28:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:28:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:28:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:28:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:28:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:28:56 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:28:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 01:29:02 Starting bucket ops to source bucket
2021/12/17 01:29:07 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:29:12 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:29:17 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:29:17 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:29:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:29:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:29:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:29:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:29:23 Rebalance progress: 12.5
2021/12/17 01:29:28 Rebalance progress: 12.5
2021/12/17 01:29:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:29:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:29:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:29:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 01:29:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:29:38 Rebalance progress: 12.5
2021/12/17 01:29:43 Rebalance progress: 12.5
2021/12/17 01:29:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:29:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:29:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:29:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 01:29:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:29:54 Rebalance progress: 12.5
2021/12/17 01:29:59 Rebalance progress: 12.5
2021/12/17 01:30:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:30:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:30:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:30:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:30:09 Rebalance progress: 12.5
2021/12/17 01:30:14 Rebalance progress: 12.5
2021/12/17 01:30:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:30:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:30:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:30:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 01:30:24 Rebalance progress: 12.5
2021/12/17 01:30:29 Rebalance progress: 12.5
2021/12/17 01:30:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:30:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:30:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:30:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 01:30:39 Rebalance progress: 12.5
2021/12/17 01:30:44 Rebalance progress: 12.5
2021/12/17 01:30:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:30:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:30:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:30:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:30:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:30:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:31:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:31:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:31:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:31:15 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:31:15 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:31:15 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 01:31:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:31:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:31:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10668
2021/12/17 01:31:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:31:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:31:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:31:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10668
2021/12/17 01:31:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (190.56s)
=== RUN   TestKVForceFailover
2021/12/17 01:31:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:31:53 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:31:53 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:31:53 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 01:31:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:32:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:32:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:32:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:32:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:32:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:32:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:32:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:32:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:32:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:32:12 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:32:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 01:32:13 Starting bucket ops to source bucket
2021/12/17 01:32:18 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:32:23 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:32:28 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:32:33 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:32:33 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:32:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:32:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:32:33 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/17 01:32:53 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/17 01:32:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 01:32:59 Rebalance progress: 12.5
2021/12/17 01:33:04 Rebalance progress: 12.5
2021/12/17 01:33:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:33:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:33:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:33:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/17 01:33:14 Rebalance progress: 12.5
2021/12/17 01:33:19 Rebalance progress: 12.5
2021/12/17 01:33:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:33:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:33:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:33:24 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:33:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:33:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:33:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:33:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:33:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:33:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:33:54 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:33:54 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:33:54 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 01:33:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:34:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:34:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7054
2021/12/17 01:34:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:34:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:34:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:34:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7054
2021/12/17 01:34:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.88s)
=== RUN   TestKVHardFailoverRecovery
2021/12/17 01:34:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:34:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:34:28 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:34:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 01:34:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:34:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:34:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:34:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:34:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:34:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:34:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:34:47 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:34:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 01:34:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 01:34:58 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:35:03 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:35:08 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:35:08 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:35:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:35:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:35:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:35:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:35:13 Rebalance progress: 12.5
2021/12/17 01:35:18 Rebalance progress: 12.5
2021/12/17 01:35:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:35:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:35:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:35:23 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/17 01:36:23 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/17 01:36:28 Rebalance progress: 12.5
2021/12/17 01:36:33 Rebalance progress: 12.5
2021/12/17 01:36:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:36:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:36:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:36:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:36:43 Rebalance progress: 12.5
2021/12/17 01:36:48 Rebalance progress: 12.5
2021/12/17 01:36:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:36:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:36:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:36:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:36:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:37:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:37:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:37:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:37:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:37:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:37:24 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:37:24 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:37:24 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 01:37:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:37:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:37:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12064
2021/12/17 01:37:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:37:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:37:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:37:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12064
2021/12/17 01:37:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.47s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/17 01:37:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:37:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:37:58 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:37:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 01:37:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:38:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:38:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:38:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:38:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:38:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:38:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:38:16 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:38:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 01:38:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 01:38:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:38:32 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:38:37 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:38:37 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:38:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:38:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:38:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:38:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 01:38:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 01:38:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:38:43 Rebalance progress: 12.5
2021/12/17 01:38:48 Rebalance progress: 12.5
2021/12/17 01:38:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:38:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:38:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:38:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/17 01:38:58 Rebalance progress: 12.5
2021/12/17 01:39:03 Rebalance progress: 12.5
2021/12/17 01:39:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:39:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:39:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:39:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:39:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:39:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:39:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:39:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:39:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:39:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:39:39 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:39:39 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:39:39 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 01:39:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:39:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:39:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2540
2021/12/17 01:39:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2540
2021/12/17 01:39:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:39:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:40:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:40:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4610
2021/12/17 01:40:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (139.80s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/17 01:40:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:40:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:40:17 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:40:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 01:40:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:40:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:40:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:40:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:40:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:40:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:40:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:40:31 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:40:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 01:40:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 01:40:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:40:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:40:52 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:40:52 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:40:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:40:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:40:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:40:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:40:57 Rebalance progress: 12.5
2021/12/17 01:41:02 Rebalance progress: 12.5
2021/12/17 01:41:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:41:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:41:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:41:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 01:41:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:41:13 Rebalance progress: 12.5
2021/12/17 01:41:18 Rebalance progress: 12.5
2021/12/17 01:41:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:41:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:41:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:41:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 01:41:28 Rebalance progress: 12.5
2021/12/17 01:41:33 Rebalance progress: 12.5
2021/12/17 01:41:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:41:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:41:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:41:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:41:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:41:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:41:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:41:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:42:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:42:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:42:08 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:42:08 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:42:08 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 01:42:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:42:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:42:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1404
2021/12/17 01:42:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1404
2021/12/17 01:42:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:42:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:42:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:42:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6088
2021/12/17 01:42:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:42:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:42:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:42:42 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:42:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 01:42:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:42:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:42:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:42:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:42:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:42:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.38s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/17 01:43:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:43:06 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:43:06 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:43:06 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 01:43:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:43:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:43:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:43:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:43:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:43:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:43:24 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:43:24 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 01:43:24 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:43:24 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/17 01:43:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:43:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:43:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:43:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:43:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:43:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:43:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:43:38 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:43:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:43:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:43:39 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:43:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/17 01:43:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 01:43:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:43:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:44:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:44:00 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/17 01:44:05 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 01:44:10 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 01:44:15 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 01:44:15 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:44:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:44:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:44:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:44:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:44:21 Rebalance progress: 12.5
2021/12/17 01:44:26 Rebalance progress: 12.5
2021/12/17 01:44:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:44:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:44:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:44:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 01:44:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:44:36 Rebalance progress: 12.5
2021/12/17 01:44:41 Rebalance progress: 12.5
2021/12/17 01:44:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:44:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:44:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:44:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 01:44:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:44:51 Rebalance progress: 12.5
2021/12/17 01:44:56 Rebalance progress: 12.5
2021/12/17 01:45:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:45:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:45:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:45:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:45:07 Rebalance progress: 12.5
2021/12/17 01:45:12 Rebalance progress: 12.5
2021/12/17 01:45:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:45:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:45:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:45:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 01:45:22 Rebalance progress: 12.5
2021/12/17 01:45:27 Rebalance progress: 12.5
2021/12/17 01:45:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:45:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:45:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:45:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 01:45:37 Rebalance progress: 12.5
2021/12/17 01:45:42 Rebalance progress: 12.5
2021/12/17 01:45:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:45:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:45:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:45:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:45:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:45:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:46:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:46:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:46:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:46:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:46:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:46:22 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:46:22 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:46:22 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 01:46:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:46:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:46:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12153
2021/12/17 01:46:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:46:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:46:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:46:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12153
2021/12/17 01:46:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:46:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:46:53 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:46:58 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:47:03 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:47:08 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:47:13 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:47:18 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:47:18 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 01:47:18 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:47:18 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/17 01:47:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:47:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:47:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:47:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:47:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:47:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (275.94s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/17 01:47:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:47:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:47:42 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:47:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 01:47:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:47:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:47:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:47:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:47:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:47:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:48:00 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:48:00 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 01:48:00 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:48:00 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/17 01:48:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:48:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:48:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:48:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:48:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:48:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:48:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:48:14 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:48:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:48:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:48:15 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:48:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/17 01:48:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 01:48:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:48:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:48:36 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:48:36 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/17 01:48:41 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 01:48:46 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 01:48:51 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 01:48:51 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:48:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:48:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:48:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:48:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:48:56 Rebalance progress: 12.5
2021/12/17 01:49:01 Rebalance progress: 12.5
2021/12/17 01:49:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:49:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:49:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:49:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 01:49:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:49:12 Rebalance progress: 12.5
2021/12/17 01:49:17 Rebalance progress: 12.5
2021/12/17 01:49:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:49:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:49:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:49:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 01:49:27 Rebalance progress: 12.5
2021/12/17 01:49:32 Rebalance progress: 12.5
2021/12/17 01:49:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:49:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:49:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:49:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:49:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:49:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:49:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:49:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:50:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:50:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:50:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:50:07 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:50:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 01:50:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:50:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:50:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1306
2021/12/17 01:50:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1306
2021/12/17 01:50:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:50:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:50:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:50:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7586
2021/12/17 01:50:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:50:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:50:43 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:50:48 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:50:53 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:50:58 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:51:03 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:51:08 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 01:51:08 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 01:51:08 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:51:08 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/17 01:51:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:51:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:51:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:51:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:51:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:51:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (230.07s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/17 01:51:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:51:32 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:51:32 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:51:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 01:51:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:51:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:51:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:51:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:51:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:51:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:51:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:51:50 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:51:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 01:51:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 01:52:01 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:52:06 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:52:11 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:52:11 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:52:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:52:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:52:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:52:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:52:32 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:52:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:52:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:52:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:52:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:52:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:52:48 Rebalance progress: 12.5
2021/12/17 01:52:53 Rebalance progress: 12.5
2021/12/17 01:52:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:52:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:52:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:52:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 01:52:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:53:03 Rebalance progress: 12.5
2021/12/17 01:53:08 Rebalance progress: 12.5
2021/12/17 01:53:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:53:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:53:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:53:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 01:53:18 Rebalance progress: 12.5
2021/12/17 01:53:23 Rebalance progress: 12.5
2021/12/17 01:53:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:53:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:53:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:53:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:53:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:53:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:53:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:53:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:53:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:53:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:53:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:53:58 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:53:59 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 01:54:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:54:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:54:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3338
2021/12/17 01:54:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3338
2021/12/17 01:54:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:54:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:54:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:54:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9166
2021/12/17 01:54:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.41s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/17 01:54:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:54:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:54:37 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:54:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 01:54:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:54:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:54:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:54:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:54:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:54:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:54:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:54:56 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:54:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 01:55:02 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 01:55:07 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:55:12 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:55:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:55:17 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:55:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:55:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:55:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:55:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:55:22 Rebalance progress: 12.5
2021/12/17 01:55:27 Rebalance progress: 12.5
2021/12/17 01:55:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:55:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:55:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:55:32 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:55:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:55:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:55:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:55:47 Rebalance progress: 12.5
2021/12/17 01:55:52 Rebalance progress: 12.5
2021/12/17 01:55:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:55:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:55:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:55:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 01:55:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 01:56:03 Rebalance progress: 12.5
2021/12/17 01:56:08 Rebalance progress: 12.5
2021/12/17 01:56:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:56:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:56:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:56:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 01:56:18 Rebalance progress: 12.5
2021/12/17 01:56:23 Rebalance progress: 12.5
2021/12/17 01:56:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:56:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:56:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:56:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:56:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:56:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:56:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:56:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:56:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:56:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:56:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:56:58 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:56:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 01:57:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:57:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:57:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1158
2021/12/17 01:57:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1158
2021/12/17 01:57:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:57:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:57:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:57:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/17 01:57:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.74s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/17 01:57:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 01:57:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 01:57:37 Rebalance progress: 100
2021/12/17 01:57:37 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:58:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:58:07 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 01:58:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 01:58:13 Removing node(s): [] from the cluster
2021/12/17 01:58:13 Starting bucket ops to source bucket
2021/12/17 01:58:18 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/17 01:58:23 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:58:28 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 01:58:28 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:58:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:58:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 01:58:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 01:58:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:58:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:58:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:58:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:58:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:58:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 01:58:59 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 01:58:59 eventingSetup.go:46: No eventing-con process running
2021/12/17 01:58:59 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 01:59:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:59:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:59:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 917
2021/12/17 01:59:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 917
2021/12/17 01:59:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 01:59:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 01:59:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 01:59:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1513
2021/12/17 01:59:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.52s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12177.152s

Rebalance tests took 3 hours, 22 minutes and 59 seconds.

Testing Succeeded


Version: versions-16.12.2021-18.14.cfg
Build Log: make-16.12.2021-18.14.log
Server Log: logs-16.12.2021-18.14.tar.gz

Finished